task e2e-llm-inference-service has failed: "step-fail-if-needed" exited with code 1: Error [get-kubeconfig] Found kubeconfig secret: cluster-nthg8-admin-kubeconfig [get-kubeconfig] Wrote kubeconfig to /credentials/cluster-nthg8-kubeconfig [get-kubeconfig] Found admin password secret: cluster-nthg8-admin-password [get-kubeconfig] Retrieved username [get-kubeconfig] Wrote password to /credentials/cluster-nthg8-password [get-kubeconfig] API Server URL: https://afb35e615998e4024955f70c61cd36d7-c1814657712f161b.elb.us-east-1.amazonaws.com:6443 [get-kubeconfig] Console URL: https://console-openshift-console.apps.764fc977-3a16-4fab-8f50-575dca4ddc2c.prod.konfluxeaas.com [clone-repo] rhoaieng-76590-enable-csr [clone-repo] https://github.com/mholder6/kserve [clone-repo] Cloning into '/workspace/source'... [clone-repo] Updating files: 46% (1493/3241) Updating files: 47% (1524/3241) Updating files: 48% (1556/3241) Updating files: 49% (1589/3241) Updating files: 50% (1621/3241) Updating files: 51% (1653/3241) Updating files: 52% (1686/3241) Updating files: 53% (1718/3241) Updating files: 54% (1751/3241) Updating files: 55% (1783/3241) Updating files: 56% (1815/3241) Updating files: 57% (1848/3241) Updating files: 58% (1880/3241) Updating files: 59% (1913/3241) Updating files: 60% (1945/3241) Updating files: 61% (1978/3241) Updating files: 62% (2010/3241) Updating files: 63% (2042/3241) Updating files: 64% (2075/3241) Updating files: 65% (2107/3241) Updating files: 66% (2140/3241) Updating files: 67% (2172/3241) Updating files: 68% (2204/3241) Updating files: 69% (2237/3241) Updating files: 70% (2269/3241) Updating files: 71% (2302/3241) Updating files: 72% (2334/3241) Updating files: 73% (2366/3241) Updating files: 74% (2399/3241) Updating files: 75% (2431/3241) Updating files: 76% (2464/3241) Updating files: 77% (2496/3241) Updating files: 78% (2528/3241) Updating files: 79% (2561/3241) Updating files: 80% (2593/3241) Updating files: 81% (2626/3241) Updating files: 82% (2658/3241) Updating files: 83% (2691/3241) Updating files: 84% (2723/3241) Updating files: 85% (2755/3241) Updating files: 86% (2788/3241) Updating files: 87% (2820/3241) Updating files: 88% (2853/3241) Updating files: 89% (2885/3241) Updating files: 90% (2917/3241) Updating files: 91% (2950/3241) Updating files: 92% (2982/3241) Updating files: 93% (3015/3241) Updating files: 94% (3047/3241) Updating files: 95% (3079/3241) Updating files: 96% (3112/3241) Updating files: 97% (3144/3241) Updating files: 98% (3177/3241) Updating files: 99% (3209/3241) Updating files: 100% (3241/3241) Updating files: 100% (3241/3241), done. [e2e-llm-inference-service] + bash [e2e-llm-inference-service] + STATUS_FILE=/test-status/deploy-and-e2e-status [e2e-llm-inference-service] + echo failed [e2e-llm-inference-service] + COMPONENT_NAME=kserve-agent-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-agent-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:5fdf09a9c0638d87cd14c30135665dfbd419945575bd13b362bf05d6740238cf [e2e-llm-inference-service] + KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:5fdf09a9c0638d87cd14c30135665dfbd419945575bd13b362bf05d6740238cf [e2e-llm-inference-service] + COMPONENT_NAME=kserve-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:2f1ff91e922d7f051c460730147d020b315c81ad356c3350f48c230ac68fedf4 [e2e-llm-inference-service] + KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:2f1ff91e922d7f051c460730147d020b315c81ad356c3350f48c230ac68fedf4 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-router-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-router-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:38bb3b78a757da7f5dbbacf1c5427d6766b51b8b8e644459af396c181aac4b0a [e2e-llm-inference-service] + KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:38bb3b78a757da7f5dbbacf1c5427d6766b51b8b8e644459af396c181aac4b0a [e2e-llm-inference-service] + COMPONENT_NAME=kserve-storage-initializer-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-storage-initializer-ci '.[$component_name].image' [e2e-llm-inference-service] + export STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:766bf293cdc0c4d8490474470e0c0992ef88060d1540c60d45ae6827b1115e1c [e2e-llm-inference-service] + STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:766bf293cdc0c4d8490474470e0c0992ef88060d1540c60d45ae6827b1115e1c [e2e-llm-inference-service] + COMPONENT_NAME=odh-kserve-llmisvc-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name odh-kserve-llmisvc-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:6ec4d6e6ae8d0365e70c54ba938870a864ff897f4e1bc7bd6b26686d4811ba4f [e2e-llm-inference-service] + LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:6ec4d6e6ae8d0365e70c54ba938870a864ff897f4e1bc7bd6b26686d4811ba4f [e2e-llm-inference-service] + ./test/scripts/openshift-ci/run-e2e-tests.sh 'llminferenceservice and cluster_cpu and not autoscaling and not tracing' 2 llm-d [e2e-llm-inference-service] INFERENCE_POOL_GROUP=inference.networking.k8s.io (detected from OCP 4.21.26) [e2e-llm-inference-service] Installing on cluster [e2e-llm-inference-service] Using namespace: kserve for KServe components [e2e-llm-inference-service] SKLEARN_IMAGE=quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1803 [e2e-llm-inference-service] OPT_125M_MODEL_URI=s3://example-models/facebook/opt-125m [e2e-llm-inference-service] ERROR_404_ISVC_IMAGE=quay.io/opendatahub/error-404-isvc:odh-pr-1803 [e2e-llm-inference-service] SUCCESS_200_ISVC_IMAGE=quay.io/opendatahub/success-200-isvc:odh-pr-1803 [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Kustomize v5.8.1 to /workspace/source/bin/kustomize [e2e-llm-inference-service] v5.8.1 [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] [INFO] Installing yq v4.52.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed yq v4.52.1 to /workspace/source/bin/yq [e2e-llm-inference-service] yq (https://github.com/mikefarah/yq/) version v4.52.1 [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Installing KServe Python SDK ... [e2e-llm-inference-service] [INFO] Installing uv 0.7.8 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed uv 0.7.8 to /workspace/source/bin/uv [e2e-llm-inference-service] warning: Failed to read project metadata (No `pyproject.toml` found in current directory or any parent directory). Running `uv self version` for compatibility. This fallback will be removed in the future; pass `--preview` to force an error. [e2e-llm-inference-service] uv 0.7.8 [e2e-llm-inference-service] Creating virtual environment... [e2e-llm-inference-service] warning: virtualenv's `--clear` has no effect (uv always clears the virtual environment) [e2e-llm-inference-service] Using CPython 3.9.25 interpreter at: /usr/bin/python3 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] Using CPython 3.11.13 interpreter at: /usr/bin/python3.11 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] Resolved 274 packages in 1ms [e2e-llm-inference-service] Building kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading pandas (12.5MiB) [e2e-llm-inference-service] Downloading cryptography (4.5MiB) [e2e-llm-inference-service] Downloading pyarrow (40.1MiB) [e2e-llm-inference-service] Downloading kubernetes (1.9MiB) [e2e-llm-inference-service] Downloading pydantic-core (2.0MiB) [e2e-llm-inference-service] Downloading aiohttp (1.7MiB) [e2e-llm-inference-service] Downloading setuptools (1.2MiB) [e2e-llm-inference-service] Downloading black (1.6MiB) [e2e-llm-inference-service] Downloading mypy (17.2MiB) [e2e-llm-inference-service] Downloading botocore (12.9MiB) [e2e-llm-inference-service] Downloading grpcio (6.4MiB) [e2e-llm-inference-service] Downloading numpy (15.7MiB) [e2e-llm-inference-service] Downloading uvloop (3.8MiB) [e2e-llm-inference-service] Downloading grpcio-tools (2.5MiB) [e2e-llm-inference-service] Downloading portforward (3.9MiB) [e2e-llm-inference-service] Building timeout-sampler==1.0.3 [e2e-llm-inference-service] Building python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading aiohttp [e2e-llm-inference-service] Downloading pydantic-core [e2e-llm-inference-service] Downloading black [e2e-llm-inference-service] Downloading grpcio-tools [e2e-llm-inference-service] Downloading setuptools [e2e-llm-inference-service] Built python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading portforward [e2e-llm-inference-service] Downloading uvloop [e2e-llm-inference-service] Downloading cryptography [e2e-llm-inference-service] Downloading grpcio [e2e-llm-inference-service] Downloading kubernetes [e2e-llm-inference-service] Built timeout-sampler==1.0.3 [e2e-llm-inference-service] Downloading numpy [e2e-llm-inference-service] Built kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading pandas [e2e-llm-inference-service] Downloading botocore [e2e-llm-inference-service] Downloading pyarrow [e2e-llm-inference-service] Downloading mypy [e2e-llm-inference-service] Prepared 101 packages in 1.87s [e2e-llm-inference-service] warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance. [e2e-llm-inference-service] If the cache and target directories are on different filesystems, hardlinking may not be supported. [e2e-llm-inference-service] If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning. [e2e-llm-inference-service] Installed 101 packages in 320ms [e2e-llm-inference-service] + aiohappyeyeballs==2.6.1 [e2e-llm-inference-service] + aiohttp==3.14.1 [e2e-llm-inference-service] + aiosignal==1.4.0 [e2e-llm-inference-service] + annotated-doc==0.0.4 [e2e-llm-inference-service] + annotated-types==0.7.0 [e2e-llm-inference-service] + anyio==4.9.0 [e2e-llm-inference-service] + attrs==25.3.0 [e2e-llm-inference-service] + avro==1.12.0 [e2e-llm-inference-service] + black==24.3.0 [e2e-llm-inference-service] + boto3==1.37.35 [e2e-llm-inference-service] + botocore==1.37.35 [e2e-llm-inference-service] + cachetools==5.5.2 [e2e-llm-inference-service] + certifi==2025.1.31 [e2e-llm-inference-service] + cffi==2.0.0 [e2e-llm-inference-service] + charset-normalizer==3.4.1 [e2e-llm-inference-service] + click==8.4.2 [e2e-llm-inference-service] + cloudevents==1.11.0 [e2e-llm-inference-service] + colorama==0.4.6 [e2e-llm-inference-service] + colorlog==6.10.1 [e2e-llm-inference-service] + coverage==7.8.0 [e2e-llm-inference-service] + cryptography==49.0.0 [e2e-llm-inference-service] + deprecation==2.1.0 [e2e-llm-inference-service] + durationpy==0.9 [e2e-llm-inference-service] + execnet==2.1.1 [e2e-llm-inference-service] + fastapi==0.136.3 [e2e-llm-inference-service] + frozenlist==1.5.0 [e2e-llm-inference-service] + google-auth==2.39.0 [e2e-llm-inference-service] + grpc-interceptor==0.15.4 [e2e-llm-inference-service] + grpcio==1.78.1 [e2e-llm-inference-service] + grpcio-testing==1.78.1 [e2e-llm-inference-service] + grpcio-tools==1.78.1 [e2e-llm-inference-service] + h11==0.16.0 [e2e-llm-inference-service] + httpcore==1.0.9 [e2e-llm-inference-service] + httptools==0.6.4 [e2e-llm-inference-service] + httpx==0.27.2 [e2e-llm-inference-service] + httpx-retries==0.4.5 [e2e-llm-inference-service] + idna==3.10 [e2e-llm-inference-service] + iniconfig==2.1.0 [e2e-llm-inference-service] + jinja2==3.1.6 [e2e-llm-inference-service] + jmespath==1.0.1 [e2e-llm-inference-service] + kserve==0.20.0rc0 (from file:///workspace/source/python/kserve) [e2e-llm-inference-service] + kubernetes==32.0.1 [e2e-llm-inference-service] + markupsafe==3.0.2 [e2e-llm-inference-service] + multidict==6.4.3 [e2e-llm-inference-service] + mypy==0.991 [e2e-llm-inference-service] + mypy-extensions==1.0.0 [e2e-llm-inference-service] + numpy==2.2.4 [e2e-llm-inference-service] + oauthlib==3.2.2 [e2e-llm-inference-service] + orjson==3.11.9 [e2e-llm-inference-service] + packaging==24.2 [e2e-llm-inference-service] + pandas==2.2.3 [e2e-llm-inference-service] + pathspec==0.12.1 [e2e-llm-inference-service] + platformdirs==4.3.7 [e2e-llm-inference-service] + pluggy==1.5.0 [e2e-llm-inference-service] + portforward==0.7.1 [e2e-llm-inference-service] + prometheus-client==0.21.1 [e2e-llm-inference-service] + propcache==0.3.1 [e2e-llm-inference-service] + protobuf==6.33.5 [e2e-llm-inference-service] + psutil==5.9.8 [e2e-llm-inference-service] + pyarrow==19.0.1 [e2e-llm-inference-service] + pyasn1==0.6.3 [e2e-llm-inference-service] + pyasn1-modules==0.4.2 [e2e-llm-inference-service] + pycparser==2.22 [e2e-llm-inference-service] + pydantic==2.12.4 [e2e-llm-inference-service] + pydantic-core==2.41.5 [e2e-llm-inference-service] + pyjwt==2.13.0 [e2e-llm-inference-service] + pytest==7.4.4 [e2e-llm-inference-service] + pytest-asyncio==0.23.8 [e2e-llm-inference-service] + pytest-cov==5.0.0 [e2e-llm-inference-service] + pytest-httpx==0.30.0 [e2e-llm-inference-service] + pytest-json-report==1.5.0 [e2e-llm-inference-service] + pytest-metadata==3.1.1 [e2e-llm-inference-service] + pytest-xdist==3.6.1 [e2e-llm-inference-service] + python-dateutil==2.9.0.post0 [e2e-llm-inference-service] + python-dotenv==1.1.0 [e2e-llm-inference-service] + python-multipart==0.0.32 [e2e-llm-inference-service] + python-simple-logger==2.0.19 [e2e-llm-inference-service] + pytz==2025.2 [e2e-llm-inference-service] + pyyaml==6.0.2 [e2e-llm-inference-service] + requests==2.32.3 [e2e-llm-inference-service] + requests-oauthlib==2.0.0 [e2e-llm-inference-service] + rsa==4.9.1 [e2e-llm-inference-service] + s3transfer==0.11.4 [e2e-llm-inference-service] + setuptools==78.1.0 [e2e-llm-inference-service] + six==1.17.0 [e2e-llm-inference-service] + sniffio==1.3.1 [e2e-llm-inference-service] + starlette==1.2.1 [e2e-llm-inference-service] + tabulate==0.9.0 [e2e-llm-inference-service] + timeout-sampler==1.0.3 [e2e-llm-inference-service] + timing-asgi==0.3.1 [e2e-llm-inference-service] + tomlkit==0.13.2 [e2e-llm-inference-service] + typing-extensions==4.15.0 [e2e-llm-inference-service] + typing-inspection==0.4.2 [e2e-llm-inference-service] + tzdata==2025.2 [e2e-llm-inference-service] + urllib3==2.7.0 [e2e-llm-inference-service] + uvicorn==0.34.1 [e2e-llm-inference-service] + uvloop==0.21.0 [e2e-llm-inference-service] + watchfiles==1.0.5 [e2e-llm-inference-service] + websocket-client==1.8.0 [e2e-llm-inference-service] + websockets==15.0.1 [e2e-llm-inference-service] + yarl==1.20.0 [e2e-llm-inference-service] Audited 1 package in 46ms [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Now using project "kserve" on server "https://afb35e615998e4024955f70c61cd36d7-c1814657712f161b.elb.us-east-1.amazonaws.com:6443". [e2e-llm-inference-service] [e2e-llm-inference-service] You can add applications to this project with the 'new-app' command. For example, try: [e2e-llm-inference-service] [e2e-llm-inference-service] oc new-app rails-postgresql-example [e2e-llm-inference-service] [e2e-llm-inference-service] to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: [e2e-llm-inference-service] [e2e-llm-inference-service] kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- /agnhost serve-hostname [e2e-llm-inference-service] [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Creating namespace openshift-keda... [e2e-llm-inference-service] namespace/openshift-keda created [e2e-llm-inference-service] Namespace openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating OperatorGroup openshift-keda... [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-keda created [e2e-llm-inference-service] OperatorGroup openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating Subscription for openshift-custom-metrics-autoscaler-operator... [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-custom-metrics-autoscaler-operator created [e2e-llm-inference-service] Subscription openshift-custom-metrics-autoscaler-operator created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Waiting for openshift-custom-metrics-autoscaler-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (20/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (40/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (45/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 is ready (Phase: Succeeded). [e2e-llm-inference-service] --- [e2e-llm-inference-service] Applying KedaController custom resource... [e2e-llm-inference-service] Warning: resource kedacontrollers/keda is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] kedacontroller.keda.sh/keda configured [e2e-llm-inference-service] KedaController custom resource applied. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Allowing time for KEDA components to be provisioned by the operator ... [e2e-llm-inference-service] Waiting for KEDA Operator pod (selector: "app=keda-operator") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-operator" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-operator" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-operator" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-operator-77f648d694-7c8hf 1/1 Running 0 42s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-operator" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-operator-77f648d694-7c8hf condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-operator" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Operator pod is ready. [e2e-llm-inference-service] Waiting for KEDA Metrics API Server pod (selector: "app=keda-metrics-apiserver") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-metrics-apiserver" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-metrics-apiserver-55557df8c-flrt2 1/1 Running 0 47s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-metrics-apiserver-55557df8c-flrt2 condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Metrics API Server pod is ready. [e2e-llm-inference-service] Waiting for KEDA Webhook pod (selector: "app=keda-admission-webhooks") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-admission-webhooks" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-admission-56f9798bb4-w9sj9 1/1 Running 0 53s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-admission-56f9798bb4-w9sj9 condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Webhook pod is ready. [e2e-llm-inference-service] --- [e2e-llm-inference-service] ✅ KEDA deployment script finished successfully. [e2e-llm-inference-service] KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:2f1ff91e922d7f051c460730147d020b315c81ad356c3350f48c230ac68fedf4 [e2e-llm-inference-service] LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:6ec4d6e6ae8d0365e70c54ba938870a864ff897f4e1bc7bd6b26686d4811ba4f [e2e-llm-inference-service] KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:5fdf09a9c0638d87cd14c30135665dfbd419945575bd13b362bf05d6740238cf [e2e-llm-inference-service] KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:38bb3b78a757da7f5dbbacf1c5427d6766b51b8b8e644459af396c181aac4b0a [e2e-llm-inference-service] STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:766bf293cdc0c4d8490474470e0c0992ef88060d1540c60d45ae6827b1115e1c [e2e-llm-inference-service] Installing KServe via kustomize... [e2e-llm-inference-service] === Final params.env [e2e-llm-inference-service] kserve-controller=quay.io/opendatahub/kserve-controller@sha256:2f1ff91e922d7f051c460730147d020b315c81ad356c3350f48c230ac68fedf4 [e2e-llm-inference-service] llmisvc-controller=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:6ec4d6e6ae8d0365e70c54ba938870a864ff897f4e1bc7bd6b26686d4811ba4f [e2e-llm-inference-service] kserve-agent=quay.io/opendatahub/kserve-agent@sha256:5fdf09a9c0638d87cd14c30135665dfbd419945575bd13b362bf05d6740238cf [e2e-llm-inference-service] kserve-router=quay.io/opendatahub/kserve-router@sha256:38bb3b78a757da7f5dbbacf1c5427d6766b51b8b8e644459af396c181aac4b0a [e2e-llm-inference-service] kserve-storage-initializer=quay.io/opendatahub/kserve-storage-initializer@sha256:766bf293cdc0c4d8490474470e0c0992ef88060d1540c60d45ae6827b1115e1c [e2e-llm-inference-service] kserve-llm-d=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-inference-scheduler=quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] kserve-llm-d-routing-sidecar=quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] kserve-llm-d-uds-tokenizer=quay.io/opendatahub/llm-d-kv-cache:v0.8.0 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-1=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-2=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-1-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-2-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-1=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-2=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-1-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-2-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-1=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-2=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-1-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-2-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-1=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-2=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-1-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-2-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-latency-predictor-prediction=quay.io/opendatahub/odh-latency-predictor-prediction:odh-stable [e2e-llm-inference-service] kserve-llm-d-latency-predictor-training=quay.io/opendatahub/odh-latency-predictor-training:odh-stable [e2e-llm-inference-service] # TODO update when our changes are introduced in the official image [e2e-llm-inference-service] kube-rbac-proxy=quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3 [e2e-llm-inference-service] kserve-localmodel-controller=quay.io/opendatahub/odh-kserve-localmodel-controller:odh-master [e2e-llm-inference-service] kserve-localmodelnode-agent=quay.io/opendatahub/odh-kserve-localmodelnode-agent:odh-master [e2e-llm-inference-service] ovms-versioning-ubi-micro=registry.redhat.io/ubi9/ubi-micro@sha256:38e934147827349f2b8b11ac9c38d7be23cb8e29f128b1c46e5c7ae54a2d23cd [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterservingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterservingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for CRDs to be established... [e2e-llm-inference-service] Waiting for CRD inferenceservices.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD inferenceservices.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD llminferenceserviceconfigs.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD llminferenceserviceconfigs.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD clusterstoragecontainers.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD clusterstoragecontainers.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD datascienceclusters.datasciencecluster.opendatahub.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met [e2e-llm-inference-service] Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterservingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-latency-predictor serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tokenizer serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/clusterservingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for llmisvc-controller-manager to be ready... [e2e-llm-inference-service] Waiting for pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to be created... [e2e-llm-inference-service] Pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" found. [e2e-llm-inference-service] Current pods for -l "control-plane=llmisvc-controller-manager" in namespace "kserve": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] llmisvc-controller-manager-77cb7f7d68-fl4nw 0/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to become ready... [e2e-llm-inference-service] pod/llmisvc-controller-manager-77cb7f7d68-fl4nw condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" are ready. [e2e-llm-inference-service] Re-Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterservingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-latency-predictor serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tokenizer serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/clusterservingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Applying DSC/DSCI resources... [e2e-llm-inference-service] dscinitialization.dscinitialization.opendatahub.io/test-dsci created [e2e-llm-inference-service] datasciencecluster.datasciencecluster.opendatahub.io/test-dsc created [e2e-llm-inference-service] KServe manual installation complete [e2e-llm-inference-service] 🔧 Configuration: [e2e-llm-inference-service] KServe deployment: ❌ disabled [e2e-llm-inference-service] Kuadrant deployment: ✅ enabled [e2e-llm-inference-service] [e2e-llm-inference-service] Checking OpenShift server version...(4.21.26) [e2e-llm-inference-service] 🎯 Server version (4.21.26) is 4.19.9 or higher - continue with the script [e2e-llm-inference-service] ⏳ Installing cert-manager [e2e-llm-inference-service] namespace/cert-manager-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] Waiting for openshift-cert-manager-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (10/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 found, but not yet Succeeded (Phase: Installing). Waiting... (15/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD certificates.cert-manager.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD certificates.cert-manager.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io condition met [e2e-llm-inference-service] ✅ Cert-manager installed [e2e-llm-inference-service] ⏳ Installing openshift-lws-operator [e2e-llm-inference-service] namespace/openshift-lws-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] subscription.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] Waiting for leader-worker-set CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (10/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (15/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD leaderworkersetoperators.operator.openshift.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD leaderworkersetoperators.operator.openshift.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/leaderworkersetoperators.operator.openshift.io condition met [e2e-llm-inference-service] leaderworkersetoperator.operator.openshift.io/cluster created [e2e-llm-inference-service] ⏳ waiting for openshift-lws-operator to be ready.… [e2e-llm-inference-service] Waiting for pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to be created... [e2e-llm-inference-service] Pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" found. [e2e-llm-inference-service] Current pods for -l "name=openshift-lws-operator" in namespace "openshift-lws-operator": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-lws-operator-fd8ccff4c-dqnrn 1/1 Running 0 13s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to become ready... [e2e-llm-inference-service] pod/openshift-lws-operator-fd8ccff4c-dqnrn condition met [e2e-llm-inference-service] Pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" are ready. [e2e-llm-inference-service] ✅ openshift-lws-operator installed [e2e-llm-inference-service] gatewayclass.gateway.networking.k8s.io/openshift-default created [e2e-llm-inference-service] Waiting for pod -l "app=istiod" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "app=istiod" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "app=istiod" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] istiod-openshift-gateway-5677ccb49b-hkbqw 0/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "app=istiod" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/istiod-openshift-gateway-5677ccb49b-hkbqw condition met [e2e-llm-inference-service] Pod(s) -l "app=istiod" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Creating a Gateway [e2e-llm-inference-service] Error from server (AlreadyExists): namespaces "openshift-ingress" already exists [e2e-llm-inference-service] ⏳ Creating gateway memory ConfigMap for parametersRef (2Gi) [e2e-llm-inference-service] configmap/gateway-proxy-config created [e2e-llm-inference-service] gateway.gateway.networking.k8s.io/openshift-ai-inference created [e2e-llm-inference-service] Waiting for pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c 1/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c condition met [e2e-llm-inference-service] Pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Installing RHCL(Kuadrant) operator [e2e-llm-inference-service] namespace/kuadrant-system created [e2e-llm-inference-service] subscription.operators.coreos.com/rhcl-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/kuadrant created [e2e-llm-inference-service] Waiting for rhcl-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (20/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD kuadrants.kuadrant.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD kuadrants.kuadrant.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/kuadrants.kuadrant.io condition met [e2e-llm-inference-service] Waiting for apiserver discovery /apis/kuadrant.io/v1beta1 to list kuadrants (timeout: 120s)… [e2e-llm-inference-service] Discovery for kuadrant.io/v1beta1 includes kuadrants. [e2e-llm-inference-service] ⏳ sleeping 30s after discovery (RESTMapper can trail discovery)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant created [e2e-llm-inference-service] ⏳ waiting for Kuadrant Ready (attempt 1/2, timeout 5m)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant condition met [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-56fb6f6856-n5pb8 1/1 Running 0 63s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-56fb6f6856-n5pb8 condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ⏳ waiting for authorino service to be created... [e2e-llm-inference-service] service/authorino-authorino-authorization condition met [e2e-llm-inference-service] service/authorino-authorino-authorization annotated [e2e-llm-inference-service] Warning: resource authorinos/authorino is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] authorino.operator.authorino.kuadrant.io/authorino configured [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-56fb6f6856-n5pb8 1/1 Running 0 73s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-56fb6f6856-n5pb8 condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ✅ kuadrant(authorino) installed [e2e-llm-inference-service] Setting up Jaeger (Helm) for LLMISVC tracing e2e... [e2e-llm-inference-service] ⏳ Installing Jaeger All-in-One (Helm) into namespace observability [e2e-llm-inference-service] [INFO] Installing Helm v3.16.3 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Helm v3.16.3 to /workspace/source/bin/helm [e2e-llm-inference-service] WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /credentials/cluster-nthg8-kubeconfig [e2e-llm-inference-service] WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /credentials/cluster-nthg8-kubeconfig [e2e-llm-inference-service] version.BuildInfo{Version:"v3.16.3", GitCommit:"cfd07493f46efc9debd9cc1b02a0961186df7fdf", GitTreeState:"clean", GoVersion:"go1.22.7"} [e2e-llm-inference-service] [INFO] Adding Jaeger Helm repository... [e2e-llm-inference-service] WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /credentials/cluster-nthg8-kubeconfig [e2e-llm-inference-service] WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /credentials/cluster-nthg8-kubeconfig [e2e-llm-inference-service] "jaegertracing" has been added to your repositories [e2e-llm-inference-service] [INFO] Installing Jaeger All-in-One 4.7.0... [e2e-llm-inference-service] WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /credentials/cluster-nthg8-kubeconfig [e2e-llm-inference-service] WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /credentials/cluster-nthg8-kubeconfig [e2e-llm-inference-service] NAME: jaeger [e2e-llm-inference-service] LAST DEPLOYED: Mon Jul 27 17:11:02 2026 [e2e-llm-inference-service] NAMESPACE: observability [e2e-llm-inference-service] STATUS: deployed [e2e-llm-inference-service] REVISION: 1 [e2e-llm-inference-service] TEST SUITE: None [e2e-llm-inference-service] NOTES: [e2e-llm-inference-service] ################################################################### [e2e-llm-inference-service] ### ⚠️ EXPERIMENTAL - NO STABILITY GUARANTEES ### [e2e-llm-inference-service] ### ### [e2e-llm-inference-service] ### This chart is under active development. ### [e2e-llm-inference-service] ### Breaking changes may occur in minor versions. ### [e2e-llm-inference-service] ### ### [e2e-llm-inference-service] ### See README.md for configuration details. ### [e2e-llm-inference-service] ################################################################### [e2e-llm-inference-service] [e2e-llm-inference-service] 🚀 Congratulations on successfully installing Jaeger v2.17.0 (Chart v4.7.0)! [e2e-llm-inference-service] [e2e-llm-inference-service] To access the query UI: [e2e-llm-inference-service] export POD_NAME=$(kubectl get pods --namespace observability -l "app.kubernetes.io/instance=jaeger,app.kubernetes.io/component=all-in-one" -o jsonpath="{.items[0].metadata.name}") [e2e-llm-inference-service] kubectl port-forward --namespace observability $POD_NAME 16686:16686 --address 0.0.0.0 & [e2e-llm-inference-service] Visit http://127.0.0.1:16686/ [e2e-llm-inference-service] [SUCCESS] Successfully installed Jaeger All-in-One via Helm [e2e-llm-inference-service] [INFO] Waiting for pods with label 'app.kubernetes.io/name=jaeger' in namespace 'observability' to be created... [e2e-llm-inference-service] [INFO] Found 1 pod(s) with label 'app.kubernetes.io/name=jaeger' [e2e-llm-inference-service] [INFO] Waiting for pods with label 'app.kubernetes.io/name=jaeger' in namespace 'observability' to be ready... [e2e-llm-inference-service] pod/jaeger-6bbd7d4c9d-pqfkp condition met [e2e-llm-inference-service] [SUCCESS] Pods with label 'app.kubernetes.io/name=jaeger' in namespace 'observability' are ready! [e2e-llm-inference-service] [SUCCESS] Jaeger is ready! [e2e-llm-inference-service] ⏳ Verifying Jaeger Service ports (OTLP 4317, Query 16686)... [e2e-llm-inference-service] ✅ Jaeger (Helm) ready — OTLP http://jaeger.observability.svc.cluster.local:4317 [e2e-llm-inference-service] Patching ingress domain... [e2e-llm-inference-service] configmap/inferenceservice-config patched [e2e-llm-inference-service] pod "kserve-controller-manager-6ccfdc6bb8-mzzcx" deleted [e2e-llm-inference-service] Waiting for kserve-controller-manager to be ready... [e2e-llm-inference-service] pod/kserve-controller-manager-6ccfdc6bb8-tsbd7 condition met [e2e-llm-inference-service] Installing ODH Model Controller manually... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/accounts.nim.opendatahub.io created [e2e-llm-inference-service] serviceaccount/model-serving-api created [e2e-llm-inference-service] serviceaccount/odh-model-controller created [e2e-llm-inference-service] role.rbac.authorization.k8s.io/leader-election-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-editor-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-viewer-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-prometheus-k8s created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/metrics-reader created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/odh-model-controller-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/proxy-role created [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/odh-model-controller-rolebinding-opendatahub created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/proxy-rolebinding created [e2e-llm-inference-service] configmap/odh-model-controller-parameters created [e2e-llm-inference-service] service/model-serving-api created [e2e-llm-inference-service] service/odh-model-controller-metrics-service created [e2e-llm-inference-service] service/odh-model-controller-webhook-service created [e2e-llm-inference-service] deployment.apps/model-serving-api created [e2e-llm-inference-service] deployment.apps/odh-model-controller created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/model-serving-api-metrics created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/odh-model-controller-metrics-monitor created [e2e-llm-inference-service] template.template.openshift.io/autogluon-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/guardrails-detector-huggingface-serving-template created [e2e-llm-inference-service] template.template.openshift.io/kserve-ovms created [e2e-llm-inference-service] template.template.openshift.io/mlserver-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/mlserver-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-2 created [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/validating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] Waiting for deployment "odh-model-controller" rollout to finish: 0 of 1 updated replicas are available... [e2e-llm-inference-service] deployment "odh-model-controller" successfully rolled out [e2e-llm-inference-service] networkpolicy.networking.k8s.io/allow-all created [e2e-llm-inference-service] KServe setup complete (namespace: kserve) [e2e-llm-inference-service] Add testing models to SeaweedFS S3 storage ... [e2e-llm-inference-service] Waiting for SeaweedFS deployment to be ready... [e2e-llm-inference-service] deployment "seaweedfs" successfully rolled out [e2e-llm-inference-service] S3 init job not completed, re-creating... [e2e-llm-inference-service] job.batch/s3-init replaced [e2e-llm-inference-service] Waiting for S3 init job to complete... [e2e-llm-inference-service] job.batch/s3-init condition met [e2e-llm-inference-service] Prepare CI namespace and install ServingRuntimes [e2e-llm-inference-service] Setting up CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Tearing down CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Namespace kserve-ci-e2e-test does not exist, skipping deletion [e2e-llm-inference-service] CI namespace teardown complete [e2e-llm-inference-service] Creating namespace kserve-ci-e2e-test [e2e-llm-inference-service] namespace/kserve-ci-e2e-test created [e2e-llm-inference-service] Applying S3 artifact secret [e2e-llm-inference-service] secret/mlpipeline-s3-artifact created [e2e-llm-inference-service] Applying storage-config secret [e2e-llm-inference-service] secret/storage-config created [e2e-llm-inference-service] Applying SeaweedFS S3 credentials secret [e2e-llm-inference-service] secret/seaweedfs-s3-creds created [e2e-llm-inference-service] Linking seaweedfs-s3-creds to default service account [e2e-llm-inference-service] Creating odh-trusted-ca-bundle configmap [e2e-llm-inference-service] configmap/odh-trusted-ca-bundle created [e2e-llm-inference-service] Installing ServingRuntimes [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-autogluonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver-multinode created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-lgbserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-mlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-paddleserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-pmmlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-predictiveserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-sklearnserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tensorflow-serving created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-torchserve created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tritonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-vllmserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-xgbserver created [e2e-llm-inference-service] CI namespace setup complete [e2e-llm-inference-service] Setup complete [e2e-llm-inference-service] === E2E cluster / operator summary === [e2e-llm-inference-service] Client Version: 4.20.11 [e2e-llm-inference-service] Kustomize Version: v5.6.0 [e2e-llm-inference-service] Server Version: 4.21.26 [e2e-llm-inference-service] Kubernetes Version: v1.34.9 [e2e-llm-inference-service] ClusterVersion desired: 4.21.26 [e2e-llm-inference-service] ClusterVersion history (latest): 4.21.26 (Completed) [e2e-llm-inference-service] CSVs in kuadrant-system: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in openshift-keda: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] custom-metrics-autoscaler.v2.19.0-2 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in cert-manager-operator: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in openshift-lws-operator: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] leader-worker-set.v1.0.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in openshift-operators (ODH / shared operators, filtered): [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] Kuadrant / Authorino (diagnostics): [e2e-llm-inference-service] CRD kuadrants.kuadrant.io versions: v1beta1 served=true storage=true [e2e-llm-inference-service] Subscriptions in kuadrant-system: [e2e-llm-inference-service] authorino-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators authorino-operator.v1.4.2 [e2e-llm-inference-service] dns-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators dns-operator.v1.4.1 [e2e-llm-inference-service] limitador-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators limitador-operator.v1.4.1 [e2e-llm-inference-service] rhcl-operator stable redhat-operators rhcl-operator.v1.4.2 [e2e-llm-inference-service] Kuadrant CR conditions (kuadrant/kuadrant-system): [e2e-llm-inference-service] Ready=True (Ready) [e2e-llm-inference-service] KServe deployments in kserve: [e2e-llm-inference-service] kserve-controller-manager: ready=1 image=quay.io/opendatahub/kserve-controller@sha256:2f1ff91e922d7f051c460730147d020b315c81ad356c3350f48c230ac68fedf4 [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-controller@sha256:2f1ff91e922d7f051c460730147d020b315c81ad356c3350f48c230ac68fedf4 [e2e-llm-inference-service] odh-model-controller: ready=1 image=quay.io/opendatahub/odh-model-controller:odh-incubating [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-model-controller@sha256:b5c835d640b650f164ba7f36d7008760c88e2974efa65853ee167323c441580c [e2e-llm-inference-service] llmisvc-controller-manager: ready=1 image=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:6ec4d6e6ae8d0365e70c54ba938870a864ff897f4e1bc7bd6b26686d4811ba4f [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:4642fcc7fbc67317c5bb0eb888619bbdd31aade2023dd859d777d20ec3c583fc [e2e-llm-inference-service] === End E2E cluster / operator summary === [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] CA certificate extracted [e2e-llm-inference-service] REQUESTS_CA_BUNDLE=/tmp/ca.crt [e2e-llm-inference-service] Run E2E tests: llminferenceservice and cluster_cpu and not autoscaling and not tracing [e2e-llm-inference-service] Starting E2E functional tests ... [e2e-llm-inference-service] Parallelism requested for pytest is 2 [e2e-llm-inference-service] ============================= test session starts ============================== [e2e-llm-inference-service] platform linux -- Python 3.11.13, pytest-7.4.4, pluggy-1.5.0 -- /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] cachedir: .pytest_cache [e2e-llm-inference-service] metadata: {'Python': '3.11.13', 'Platform': 'Linux-5.14.0-570.122.1.el9_6.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.5.0'}, 'Plugins': {'anyio': '4.9.0', 'asyncio': '0.23.8', 'xdist': '3.6.1', 'cov': '5.0.0', 'metadata': '3.1.1', 'json-report': '1.5.0', 'httpx': '0.30.0'}, 'PLATFORM': 'el9'} [e2e-llm-inference-service] rootdir: /workspace/source/test/e2e [e2e-llm-inference-service] configfile: pytest.ini [e2e-llm-inference-service] plugins: anyio-4.9.0, asyncio-0.23.8, xdist-3.6.1, cov-5.0.0, metadata-3.1.1, json-report-1.5.0, httpx-0.30.0 [e2e-llm-inference-service] asyncio: mode=Mode.STRICT [e2e-llm-inference-service] created: 2/2 workers [e2e-llm-inference-service] 2 workers [73 items] [e2e-llm-inference-service] [e2e-llm-inference-service] scheduling tests via WorkStealingScheduling [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] 2026-07-27 17:13:39.911 6584 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] 2026-07-27 17:13:39.912 6581 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] 2026-07-27 17:15:41.734 6584 kserve.trace Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-49b5edd9 [e2e-llm-inference-service] 2026-07-27 17:15:41.734 6584 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-49b5edd9 [e2e-llm-inference-service] 2026-07-27 17:15:41.757 6584 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-27 17:15:41.757 6584 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-27 17:15:41.763 6584 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-27 17:15:41.763 6584 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] 2026-07-27 17:16:15.455 6584 kserve.trace Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-94799d44 [e2e-llm-inference-service] 2026-07-27 17:16:15.455 6584 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-94799d44 [e2e-llm-inference-service] 2026-07-27 17:16:15.476 6584 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-27 17:16:15.476 6584 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-27 17:16:15.482 6584 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-27 17:16:15.482 6584 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_finalizer_added [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_finalizer_added [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_blocked_when_referenced [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_blocked_when_referenced [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_allowed_when_unreferenced [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_allowed_when_unreferenced [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_unblocked_after_service_deleted [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_unblocked_after_service_deleted [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_prevented_by_webhook [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_prevented_by_webhook [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_blocked_by_implicit_reference [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_blocked_by_implicit_reference [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] [e2e-llm-inference-service] =================================== FAILURES =================================== [e2e-llm-inference-service] __________ test_flow_control_smoke[flow-control-concurrency-detector] __________ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] name = 'fc-concurrency-test', namespace = 'e2e-test-flow-control-smoke-1e98f46d' [e2e-llm-inference-service] version = 'v1alpha1' [e2e-llm-inference-service] [e2e-llm-inference-service] def get_llmisvc( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] name, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] version=constants.KSERVE_V1ALPHA1_VERSION, [e2e-llm-inference-service] ): [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return kserve_client.api_instance.get_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICE, [e2e-llm-inference-service] name, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1204: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-flow-control-smoke-1e98f46d' [e2e-llm-inference-service] plural = 'llminferenceservices', name = 'fc-concurrency-test' [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] [e2e-llm-inference-service] def get_namespaced_custom_object(self, group, version, namespace, plural, name, **kwargs): # noqa: E501 [e2e-llm-inference-service] """get_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Returns a namespace scoped custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.get_namespaced_custom_object(group, version, namespace, plural, name, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: the custom resource's group (required) [e2e-llm-inference-service] :param str version: the custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: the custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param str name: the custom object's name (required) [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: object [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] kwargs['_return_http_data_only'] = True [e2e-llm-inference-service] > return self.get_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, **kwargs) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:1632: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-flow-control-smoke-1e98f46d' [e2e-llm-inference-service] plural = 'llminferenceservices', name = 'fc-concurrency-test' [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] local_var_params = {'_return_http_data_only': True, 'all_params': ['group', 'version', 'namespace', 'plural', 'name', 'async_req', ...], 'auth_settings': ['BearerToken'], 'body_params': None, ...} [e2e-llm-inference-service] all_params = ['group', 'version', 'namespace', 'plural', 'name', 'async_req', ...] [e2e-llm-inference-service] key = '_return_http_data_only', val = True, collection_formats = {} [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'name': 'fc-concurrency-test', 'namespace': 'e2e-test-flow-control-smoke-1e98f46d', 'plural': 'llminferenceservices', ...} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] [e2e-llm-inference-service] def get_namespaced_custom_object_with_http_info(self, group, version, namespace, plural, name, **kwargs): # noqa: E501 [e2e-llm-inference-service] """get_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Returns a namespace scoped custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.get_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: the custom resource's group (required) [e2e-llm-inference-service] :param str version: the custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: the custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param str name: the custom object's name (required) [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: tuple(object, status_code(int), headers(HTTPHeaderDict)) [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] local_var_params = locals() [e2e-llm-inference-service] [e2e-llm-inference-service] all_params = [ [e2e-llm-inference-service] 'group', [e2e-llm-inference-service] 'version', [e2e-llm-inference-service] 'namespace', [e2e-llm-inference-service] 'plural', [e2e-llm-inference-service] 'name' [e2e-llm-inference-service] ] [e2e-llm-inference-service] all_params.extend( [e2e-llm-inference-service] [ [e2e-llm-inference-service] 'async_req', [e2e-llm-inference-service] '_return_http_data_only', [e2e-llm-inference-service] '_preload_content', [e2e-llm-inference-service] '_request_timeout' [e2e-llm-inference-service] ] [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] for key, val in six.iteritems(local_var_params['kwargs']): [e2e-llm-inference-service] if key not in all_params: [e2e-llm-inference-service] raise ApiTypeError( [e2e-llm-inference-service] "Got an unexpected keyword argument '%s'" [e2e-llm-inference-service] " to method get_namespaced_custom_object" % key [e2e-llm-inference-service] ) [e2e-llm-inference-service] local_var_params[key] = val [e2e-llm-inference-service] del local_var_params['kwargs'] [e2e-llm-inference-service] # verify the required parameter 'group' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('group' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['group'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `group` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'version' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('version' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['version'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `version` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'namespace' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('namespace' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['namespace'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `namespace` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'plural' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('plural' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['plural'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `plural` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'name' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('name' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['name'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `name` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] collection_formats = {} [e2e-llm-inference-service] [e2e-llm-inference-service] path_params = {} [e2e-llm-inference-service] if 'group' in local_var_params: [e2e-llm-inference-service] path_params['group'] = local_var_params['group'] # noqa: E501 [e2e-llm-inference-service] if 'version' in local_var_params: [e2e-llm-inference-service] path_params['version'] = local_var_params['version'] # noqa: E501 [e2e-llm-inference-service] if 'namespace' in local_var_params: [e2e-llm-inference-service] path_params['namespace'] = local_var_params['namespace'] # noqa: E501 [e2e-llm-inference-service] if 'plural' in local_var_params: [e2e-llm-inference-service] path_params['plural'] = local_var_params['plural'] # noqa: E501 [e2e-llm-inference-service] if 'name' in local_var_params: [e2e-llm-inference-service] path_params['name'] = local_var_params['name'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] [e2e-llm-inference-service] header_params = {} [e2e-llm-inference-service] [e2e-llm-inference-service] form_params = [] [e2e-llm-inference-service] local_var_files = {} [e2e-llm-inference-service] [e2e-llm-inference-service] body_params = None [e2e-llm-inference-service] # HTTP header `Accept` [e2e-llm-inference-service] header_params['Accept'] = self.api_client.select_header_accept( [e2e-llm-inference-service] ['application/json']) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] # Authentication setting [e2e-llm-inference-service] auth_settings = ['BearerToken'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] > return self.api_client.call_api( [e2e-llm-inference-service] '/apis/{group}/{version}/namespaces/{namespace}/{plural}/{name}', 'GET', [e2e-llm-inference-service] path_params, [e2e-llm-inference-service] query_params, [e2e-llm-inference-service] header_params, [e2e-llm-inference-service] body=body_params, [e2e-llm-inference-service] post_params=form_params, [e2e-llm-inference-service] files=local_var_files, [e2e-llm-inference-service] response_type='object', # noqa: E501 [e2e-llm-inference-service] auth_settings=auth_settings, [e2e-llm-inference-service] async_req=local_var_params.get('async_req'), [e2e-llm-inference-service] _return_http_data_only=local_var_params.get('_return_http_data_only'), # noqa: E501 [e2e-llm-inference-service] _preload_content=local_var_params.get('_preload_content', True), [e2e-llm-inference-service] _request_timeout=local_var_params.get('_request_timeout'), [e2e-llm-inference-service] collection_formats=collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:1739: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/{group}/{version}/namespaces/{namespace}/{plural}/{name}' [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'name': 'fc-concurrency-test', 'namespace': 'e2e-test-flow-control-smoke-1e98f46d', 'plural': 'llminferenceservices', ...} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], async_req = None, _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def call_api(self, resource_path, method, [e2e-llm-inference-service] path_params=None, query_params=None, header_params=None, [e2e-llm-inference-service] body=None, post_params=None, files=None, [e2e-llm-inference-service] response_type=None, auth_settings=None, async_req=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] """Makes the HTTP request (synchronous) and returns deserialized data. [e2e-llm-inference-service] [e2e-llm-inference-service] To make an async_req request, set the async_req parameter. [e2e-llm-inference-service] [e2e-llm-inference-service] :param resource_path: Path to method endpoint. [e2e-llm-inference-service] :param method: Method to call. [e2e-llm-inference-service] :param path_params: Path parameters in the url. [e2e-llm-inference-service] :param query_params: Query parameters in the url. [e2e-llm-inference-service] :param header_params: Header parameters to be [e2e-llm-inference-service] placed in the request header. [e2e-llm-inference-service] :param body: Request body. [e2e-llm-inference-service] :param post_params dict: Request post form parameters, [e2e-llm-inference-service] for `application/x-www-form-urlencoded`, `multipart/form-data`. [e2e-llm-inference-service] :param auth_settings list: Auth Settings names for the request. [e2e-llm-inference-service] :param response: Response data type. [e2e-llm-inference-service] :param files dict: key -> filename, value -> filepath, [e2e-llm-inference-service] for `multipart/form-data`. [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param collection_formats: dict of collection formats for path, query, [e2e-llm-inference-service] header, and post parameters. [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: [e2e-llm-inference-service] If async_req parameter is True, [e2e-llm-inference-service] the request will be called asynchronously. [e2e-llm-inference-service] The method will return the request thread. [e2e-llm-inference-service] If parameter async_req is False or missing, [e2e-llm-inference-service] then the method will return the response directly. [e2e-llm-inference-service] """ [e2e-llm-inference-service] if not async_req: [e2e-llm-inference-service] > return self.__call_api(resource_path, method, [e2e-llm-inference-service] path_params, query_params, header_params, [e2e-llm-inference-service] body, post_params, files, [e2e-llm-inference-service] response_type, auth_settings, [e2e-llm-inference-service] _return_http_data_only, collection_formats, [e2e-llm-inference-service] _preload_content, _request_timeout, _host) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:348: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-flow-control-smoke-1e98f46d/llminferenceservices/fc-concurrency-test' [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] path_params = [('group', 'serving.kserve.io'), ('version', 'v1alpha1'), ('namespace', 'e2e-test-flow-control-smoke-1e98f46d'), ('plural', 'llminferenceservices'), ('name', 'fc-concurrency-test')] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def __call_api( [e2e-llm-inference-service] self, resource_path, method, path_params=None, [e2e-llm-inference-service] query_params=None, header_params=None, body=None, post_params=None, [e2e-llm-inference-service] files=None, response_type=None, auth_settings=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] [e2e-llm-inference-service] config = self.configuration [e2e-llm-inference-service] [e2e-llm-inference-service] # header parameters [e2e-llm-inference-service] header_params = header_params or {} [e2e-llm-inference-service] header_params.update(self.default_headers) [e2e-llm-inference-service] if self.cookie: [e2e-llm-inference-service] header_params['Cookie'] = self.cookie [e2e-llm-inference-service] if header_params: [e2e-llm-inference-service] header_params = self.sanitize_for_serialization(header_params) [e2e-llm-inference-service] header_params = dict(self.parameters_to_tuples(header_params, [e2e-llm-inference-service] collection_formats)) [e2e-llm-inference-service] [e2e-llm-inference-service] # path parameters [e2e-llm-inference-service] if path_params: [e2e-llm-inference-service] path_params = self.sanitize_for_serialization(path_params) [e2e-llm-inference-service] path_params = self.parameters_to_tuples(path_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] for k, v in path_params: [e2e-llm-inference-service] # specified safe chars, encode everything [e2e-llm-inference-service] resource_path = resource_path.replace( [e2e-llm-inference-service] '{%s}' % k, [e2e-llm-inference-service] quote(str(v), safe=config.safe_chars_for_path_param) [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # query parameters [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] query_params = self.sanitize_for_serialization(query_params) [e2e-llm-inference-service] query_params = self.parameters_to_tuples(query_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] # post parameters [e2e-llm-inference-service] if post_params or files: [e2e-llm-inference-service] post_params = post_params if post_params else [] [e2e-llm-inference-service] post_params = self.sanitize_for_serialization(post_params) [e2e-llm-inference-service] post_params = self.parameters_to_tuples(post_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] post_params.extend(self.files_parameters(files)) [e2e-llm-inference-service] [e2e-llm-inference-service] # auth setting [e2e-llm-inference-service] self.update_params_for_auth(header_params, query_params, auth_settings) [e2e-llm-inference-service] [e2e-llm-inference-service] # body [e2e-llm-inference-service] if body: [e2e-llm-inference-service] body = self.sanitize_for_serialization(body) [e2e-llm-inference-service] [e2e-llm-inference-service] # request url [e2e-llm-inference-service] if _host is None: [e2e-llm-inference-service] url = self.configuration.host + resource_path [e2e-llm-inference-service] else: [e2e-llm-inference-service] # use server/host defined in path or operation instead [e2e-llm-inference-service] url = _host + resource_path [e2e-llm-inference-service] [e2e-llm-inference-service] # perform request and return response [e2e-llm-inference-service] > response_data = self.request( [e2e-llm-inference-service] method, url, query_params=query_params, headers=header_params, [e2e-llm-inference-service] post_params=post_params, body=body, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:180: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] url = 'https://afb35e615998e4024955f70c61cd36d7-c1814657712f161b.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-flow-control-smoke-1e98f46d/llminferenceservices/fc-concurrency-test' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] post_params = [], body = None, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] post_params=None, body=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Makes the HTTP request using RESTClient.""" [e2e-llm-inference-service] if method == "GET": [e2e-llm-inference-service] > return self.rest_client.GET(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:373: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] url = 'https://afb35e615998e4024955f70c61cd36d7-c1814657712f161b.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-flow-control-smoke-1e98f46d/llminferenceservices/fc-concurrency-test' [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] query_params = [], _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def GET(self, url, headers=None, query_params=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] > return self.request("GET", url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] query_params=query_params) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:244: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] url = 'https://afb35e615998e4024955f70c61cd36d7-c1814657712f161b.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-flow-control-smoke-1e98f46d/llminferenceservices/fc-concurrency-test' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] body=None, post_params=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Perform requests. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: http request method [e2e-llm-inference-service] :param url: http request url [e2e-llm-inference-service] :param query_params: query parameters in the url [e2e-llm-inference-service] :param headers: http request headers [e2e-llm-inference-service] :param body: request json body, for `application/json` [e2e-llm-inference-service] :param post_params: request post parameters, [e2e-llm-inference-service] `application/x-www-form-urlencoded` [e2e-llm-inference-service] and `multipart/form-data` [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] """ [e2e-llm-inference-service] method = method.upper() [e2e-llm-inference-service] assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT', [e2e-llm-inference-service] 'PATCH', 'OPTIONS'] [e2e-llm-inference-service] [e2e-llm-inference-service] if post_params and body: [e2e-llm-inference-service] raise ApiValueError( [e2e-llm-inference-service] "body parameter cannot be used with post_params parameter." [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] post_params = post_params or {} [e2e-llm-inference-service] headers = headers or {} [e2e-llm-inference-service] [e2e-llm-inference-service] timeout = None [e2e-llm-inference-service] if _request_timeout: [e2e-llm-inference-service] if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)): # noqa: E501,F821 [e2e-llm-inference-service] timeout = urllib3.Timeout(total=_request_timeout) [e2e-llm-inference-service] elif (isinstance(_request_timeout, tuple) and [e2e-llm-inference-service] len(_request_timeout) == 2): [e2e-llm-inference-service] timeout = urllib3.Timeout( [e2e-llm-inference-service] connect=_request_timeout[0], read=_request_timeout[1]) [e2e-llm-inference-service] [e2e-llm-inference-service] if 'Content-Type' not in headers: [e2e-llm-inference-service] headers['Content-Type'] = 'application/json' [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE` [e2e-llm-inference-service] if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']: [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] url += '?' + urlencode(query_params) [e2e-llm-inference-service] if (re.search('json', headers['Content-Type'], re.IGNORECASE) or [e2e-llm-inference-service] headers['Content-Type'] == 'application/apply-patch+yaml'): [e2e-llm-inference-service] if headers['Content-Type'] == 'application/json-patch+json': [e2e-llm-inference-service] if not isinstance(body, list): [e2e-llm-inference-service] headers['Content-Type'] = \ [e2e-llm-inference-service] 'application/strategic-merge-patch+json' [e2e-llm-inference-service] request_body = None [e2e-llm-inference-service] if body is not None: [e2e-llm-inference-service] request_body = json.dumps(body) [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'application/x-www-form-urlencoded': # noqa: E501 [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=False, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'multipart/form-data': [e2e-llm-inference-service] # must del headers['Content-Type'], or the correct [e2e-llm-inference-service] # Content-Type which generated by urllib3 will be [e2e-llm-inference-service] # overwritten. [e2e-llm-inference-service] del headers['Content-Type'] [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=True, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] # Pass a `string` parameter directly in the body to support [e2e-llm-inference-service] # other content types than Json when `body` argument is [e2e-llm-inference-service] # provided in serialized form [e2e-llm-inference-service] elif isinstance(body, str) or isinstance(body, bytes): [e2e-llm-inference-service] request_body = body [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] else: [e2e-llm-inference-service] # Cannot generate the request from given parameters [e2e-llm-inference-service] msg = """Cannot prepare a request message for provided [e2e-llm-inference-service] arguments. Please check that your arguments match [e2e-llm-inference-service] declared content type.""" [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] # For `GET`, `HEAD` [e2e-llm-inference-service] else: [e2e-llm-inference-service] r = self.pool_manager.request(method, url, [e2e-llm-inference-service] fields=query_params, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] except urllib3.exceptions.SSLError as e: [e2e-llm-inference-service] msg = "{0}\n{1}".format(type(e).__name__, str(e)) [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] [e2e-llm-inference-service] if _preload_content: [e2e-llm-inference-service] r = RESTResponse(r) [e2e-llm-inference-service] [e2e-llm-inference-service] # In the python 3, the response.data is bytes. [e2e-llm-inference-service] # we need to decode it to string. [e2e-llm-inference-service] if six.PY3: [e2e-llm-inference-service] r.data = r.data.decode('utf8') [e2e-llm-inference-service] [e2e-llm-inference-service] # log response body [e2e-llm-inference-service] logger.debug("response body: %s", r.data) [e2e-llm-inference-service] [e2e-llm-inference-service] if not 200 <= r.status <= 299: [e2e-llm-inference-service] > raise ApiException(http_resp=r) [e2e-llm-inference-service] E kubernetes.client.exceptions.ApiException: (500) [e2e-llm-inference-service] E Reason: Internal Server Error [e2e-llm-inference-service] E HTTP response headers: HTTPHeaderDict({'Audit-Id': 'bffee7cd-8ca9-463f-9036-76650957c555', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'bf546a5c-3f60-4261-9b3f-c3b9e58fb8c8', 'X-Kubernetes-Pf-Prioritylevel-Uid': '39b8073e-7e81-4403-a6ca-a29718cbfcdd', 'Date': 'Mon, 27 Jul 2026 17:15:41 GMT', 'Content-Length': '264'}) [e2e-llm-inference-service] E HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"conversion webhook for serving.kserve.io/v1alpha2, Kind=LLMInferenceService failed: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/convert?timeout=30s\": EOF","code":500} [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:238: ApiException [e2e-llm-inference-service] [e2e-llm-inference-service] The above exception was the direct cause of the following exception: [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'scheduler-flow-control-concurrency-detector', 'workload-llmd-simulator'], promp... {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] flow_control_auth = None [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-flow-control-round-robin", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="fc-smoke-test", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.flow_control, [e2e-llm-inference-service] ], [e2e-llm-inference-service] id="flow-control-utilization-detector", [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-flow-control-concurrency-detector", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="fc-concurrency-test", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.flow_control, [e2e-llm-inference-service] ], [e2e-llm-inference-service] id="flow-control-concurrency-detector", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=True, [e2e-llm-inference-service] ) [e2e-llm-inference-service] def test_flow_control_smoke(test_case: TestCase, flow_control_auth): [e2e-llm-inference-service] """Verify that the EPP boots and serves traffic with flow control enabled. [e2e-llm-inference-service] [e2e-llm-inference-service] Sends requests with different fairness IDs, InferenceObjective headers, and [e2e-llm-inference-service] default headers. When a downstream auth provider is available (via the [e2e-llm-inference-service] flow_control_auth fixture), also verifies the auth -> flow control pipeline. [e2e-llm-inference-service] """ [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] [e2e-llm-inference-service] if flow_control_auth: [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] test_case.llm_service.metadata.annotations.update( [e2e-llm-inference-service] flow_control_auth.get("annotations", {}) [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py:122: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...curre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-27T17:14:56.368494', start_time = 1785172496.3687656 [e2e-llm-inference-service] duration = 44.869563817977905, timestamp_end = '2026-07-27T17:15:41.238334' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....ol-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f4e071f8680> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] > out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1339: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] name = 'fc-concurrency-test', namespace = 'e2e-test-flow-control-smoke-1e98f46d' [e2e-llm-inference-service] version = 'v1alpha1' [e2e-llm-inference-service] [e2e-llm-inference-service] def get_llmisvc( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] name, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] version=constants.KSERVE_V1ALPHA1_VERSION, [e2e-llm-inference-service] ): [e2e-llm-inference-service] try: [e2e-llm-inference-service] return kserve_client.api_instance.get_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICE, [e2e-llm-inference-service] name, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except client.rest.ApiException as e: [e2e-llm-inference-service] > raise RuntimeError( [e2e-llm-inference-service] f"❌ Exception when calling CustomObjectsApi->" [e2e-llm-inference-service] f"get_namespaced_custom_object for LLMInferenceService: {e}" [e2e-llm-inference-service] ) from e [e2e-llm-inference-service] E RuntimeError: ❌ Exception when calling CustomObjectsApi->get_namespaced_custom_object for LLMInferenceService: (500) [e2e-llm-inference-service] E Reason: Internal Server Error [e2e-llm-inference-service] E HTTP response headers: HTTPHeaderDict({'Audit-Id': 'bffee7cd-8ca9-463f-9036-76650957c555', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'bf546a5c-3f60-4261-9b3f-c3b9e58fb8c8', 'X-Kubernetes-Pf-Prioritylevel-Uid': '39b8073e-7e81-4403-a6ca-a29718cbfcdd', 'Date': 'Mon, 27 Jul 2026 17:15:41 GMT', 'Content-Length': '264'}) [e2e-llm-inference-service] E HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"conversion webhook for serving.kserve.io/v1alpha2, Kind=LLMInferenceService failed: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/convert?timeout=30s\": EOF","code":500} [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1212: RuntimeError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-flow-control-smoke-1e98f46d [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-flow-control-smoke-1e98f46d [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-flow-control-smoke-1e98f46d [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-flow-control-smoke-1e98f46d with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-flow-control-smoke-1e98f46d [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-flow-control-smoke-1e98f46d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-fc-concurrency-t-25c29f3a in namespace e2e-test-flow-control-smoke-1e98f46d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-fc-concurrency-t-25c29f3a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-fc-concurrency-t-25c29f3a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scheduler-flow-control-concurre-34e50aeb in namespace e2e-test-flow-control-smoke-1e98f46d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scheduler-flow-control-concurre-34e50aeb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scheduler-flow-control-concurre-34e50aeb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-fc-conc-1c5a0c75 in namespace e2e-test-flow-control-smoke-1e98f46d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-fc-conc-1c5a0c75 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-fc-conc-1c5a0c75 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-27T17:14:56.314980] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-flow-control-smoke-1e98f46d', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-27T17:14:56.368389] end - ✅ in 0.053s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-27T17:14:56.368494] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-flow-control-smoke-1e98f46d', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:15:35Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:15:35Z', 'message': 'Inference Pool e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:15:35Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:15:35Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:15:35Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:15:35Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:15:35Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:15:35Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-27T17:15:41.238334] end - ❌ 44.870s: ❌ Exception when calling CustomObjectsApi->get_namespaced_custom_object for LLMInferenceService: (500) [e2e-llm-inference-service] Reason: Internal Server Error [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': 'bffee7cd-8ca9-463f-9036-76650957c555', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'bf546a5c-3f60-4261-9b3f-c3b9e58fb8c8', 'X-Kubernetes-Pf-Prioritylevel-Uid': '39b8073e-7e81-4403-a6ca-a29718cbfcdd', 'Date': 'Mon, 27 Jul 2026 17:15:41 GMT', 'Content-Length': '264'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"conversion webhook for serving.kserve.io/v1alpha2, Kind=LLMInferenceService failed: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/convert?timeout=30s\": EOF","code":500} [e2e-llm-inference-service] [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_flow_control.py:171 [router-managed-scheduler-flow-control-concurrency-detector-workload-llmd-simulator] Failed: ❌ Exception when calling CustomObjectsApi->get_namespaced_custom_object for LLMInferenceService: (500) [e2e-llm-inference-service] Reason: Internal Server Error [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': 'bffee7cd-8ca9-463f-9036-76650957c555', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'bf546a5c-3f60-4261-9b3f-c3b9e58fb8c8', 'X-Kubernetes-Pf-Prioritylevel-Uid': '39b8073e-7e81-4403-a6ca-a29718cbfcdd', 'Date': 'Mon, 27 Jul 2026 17:15:41 GMT', 'Content-Length': '264'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"conversion webhook for serving.kserve.io/v1alpha2, Kind=LLMInferenceService failed: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/convert?timeout=30s\": EOF","code":500} [e2e-llm-inference-service] [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-flow-control-smoke-1e98f46d (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_inference_service[router-no-scheduler-workload-single-cpu-model-fb-opt-125m] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] name = 'llmisvc-model-fb-opt-125m-route-3c960099' [e2e-llm-inference-service] namespace = 'e2e-test-llm-inference-service-fac1c4ae', version = 'v1alpha1' [e2e-llm-inference-service] [e2e-llm-inference-service] def get_llmisvc( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] name, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] version=constants.KSERVE_V1ALPHA1_VERSION, [e2e-llm-inference-service] ): [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return kserve_client.api_instance.get_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICE, [e2e-llm-inference-service] name, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1204: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-inference-service-fac1c4ae' [e2e-llm-inference-service] plural = 'llminferenceservices' [e2e-llm-inference-service] name = 'llmisvc-model-fb-opt-125m-route-3c960099' [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] [e2e-llm-inference-service] def get_namespaced_custom_object(self, group, version, namespace, plural, name, **kwargs): # noqa: E501 [e2e-llm-inference-service] """get_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Returns a namespace scoped custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.get_namespaced_custom_object(group, version, namespace, plural, name, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: the custom resource's group (required) [e2e-llm-inference-service] :param str version: the custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: the custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param str name: the custom object's name (required) [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: object [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] kwargs['_return_http_data_only'] = True [e2e-llm-inference-service] > return self.get_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, **kwargs) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:1632: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-inference-service-fac1c4ae' [e2e-llm-inference-service] plural = 'llminferenceservices' [e2e-llm-inference-service] name = 'llmisvc-model-fb-opt-125m-route-3c960099' [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] local_var_params = {'_return_http_data_only': True, 'all_params': ['group', 'version', 'namespace', 'plural', 'name', 'async_req', ...], 'auth_settings': ['BearerToken'], 'body_params': None, ...} [e2e-llm-inference-service] all_params = ['group', 'version', 'namespace', 'plural', 'name', 'async_req', ...] [e2e-llm-inference-service] key = '_return_http_data_only', val = True, collection_formats = {} [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'name': 'llmisvc-model-fb-opt-125m-route-3c960099', 'namespace': 'e2e-test-llm-inference-service-fac1c4ae', 'plural': 'llminferenceservices', ...} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] [e2e-llm-inference-service] def get_namespaced_custom_object_with_http_info(self, group, version, namespace, plural, name, **kwargs): # noqa: E501 [e2e-llm-inference-service] """get_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Returns a namespace scoped custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.get_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: the custom resource's group (required) [e2e-llm-inference-service] :param str version: the custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: the custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param str name: the custom object's name (required) [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: tuple(object, status_code(int), headers(HTTPHeaderDict)) [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] local_var_params = locals() [e2e-llm-inference-service] [e2e-llm-inference-service] all_params = [ [e2e-llm-inference-service] 'group', [e2e-llm-inference-service] 'version', [e2e-llm-inference-service] 'namespace', [e2e-llm-inference-service] 'plural', [e2e-llm-inference-service] 'name' [e2e-llm-inference-service] ] [e2e-llm-inference-service] all_params.extend( [e2e-llm-inference-service] [ [e2e-llm-inference-service] 'async_req', [e2e-llm-inference-service] '_return_http_data_only', [e2e-llm-inference-service] '_preload_content', [e2e-llm-inference-service] '_request_timeout' [e2e-llm-inference-service] ] [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] for key, val in six.iteritems(local_var_params['kwargs']): [e2e-llm-inference-service] if key not in all_params: [e2e-llm-inference-service] raise ApiTypeError( [e2e-llm-inference-service] "Got an unexpected keyword argument '%s'" [e2e-llm-inference-service] " to method get_namespaced_custom_object" % key [e2e-llm-inference-service] ) [e2e-llm-inference-service] local_var_params[key] = val [e2e-llm-inference-service] del local_var_params['kwargs'] [e2e-llm-inference-service] # verify the required parameter 'group' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('group' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['group'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `group` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'version' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('version' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['version'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `version` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'namespace' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('namespace' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['namespace'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `namespace` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'plural' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('plural' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['plural'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `plural` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'name' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('name' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['name'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `name` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] collection_formats = {} [e2e-llm-inference-service] [e2e-llm-inference-service] path_params = {} [e2e-llm-inference-service] if 'group' in local_var_params: [e2e-llm-inference-service] path_params['group'] = local_var_params['group'] # noqa: E501 [e2e-llm-inference-service] if 'version' in local_var_params: [e2e-llm-inference-service] path_params['version'] = local_var_params['version'] # noqa: E501 [e2e-llm-inference-service] if 'namespace' in local_var_params: [e2e-llm-inference-service] path_params['namespace'] = local_var_params['namespace'] # noqa: E501 [e2e-llm-inference-service] if 'plural' in local_var_params: [e2e-llm-inference-service] path_params['plural'] = local_var_params['plural'] # noqa: E501 [e2e-llm-inference-service] if 'name' in local_var_params: [e2e-llm-inference-service] path_params['name'] = local_var_params['name'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] [e2e-llm-inference-service] header_params = {} [e2e-llm-inference-service] [e2e-llm-inference-service] form_params = [] [e2e-llm-inference-service] local_var_files = {} [e2e-llm-inference-service] [e2e-llm-inference-service] body_params = None [e2e-llm-inference-service] # HTTP header `Accept` [e2e-llm-inference-service] header_params['Accept'] = self.api_client.select_header_accept( [e2e-llm-inference-service] ['application/json']) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] # Authentication setting [e2e-llm-inference-service] auth_settings = ['BearerToken'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] > return self.api_client.call_api( [e2e-llm-inference-service] '/apis/{group}/{version}/namespaces/{namespace}/{plural}/{name}', 'GET', [e2e-llm-inference-service] path_params, [e2e-llm-inference-service] query_params, [e2e-llm-inference-service] header_params, [e2e-llm-inference-service] body=body_params, [e2e-llm-inference-service] post_params=form_params, [e2e-llm-inference-service] files=local_var_files, [e2e-llm-inference-service] response_type='object', # noqa: E501 [e2e-llm-inference-service] auth_settings=auth_settings, [e2e-llm-inference-service] async_req=local_var_params.get('async_req'), [e2e-llm-inference-service] _return_http_data_only=local_var_params.get('_return_http_data_only'), # noqa: E501 [e2e-llm-inference-service] _preload_content=local_var_params.get('_preload_content', True), [e2e-llm-inference-service] _request_timeout=local_var_params.get('_request_timeout'), [e2e-llm-inference-service] collection_formats=collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:1739: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/{group}/{version}/namespaces/{namespace}/{plural}/{name}' [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'name': 'llmisvc-model-fb-opt-125m-route-3c960099', 'namespace': 'e2e-test-llm-inference-service-fac1c4ae', 'plural': 'llminferenceservices', ...} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], async_req = None, _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def call_api(self, resource_path, method, [e2e-llm-inference-service] path_params=None, query_params=None, header_params=None, [e2e-llm-inference-service] body=None, post_params=None, files=None, [e2e-llm-inference-service] response_type=None, auth_settings=None, async_req=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] """Makes the HTTP request (synchronous) and returns deserialized data. [e2e-llm-inference-service] [e2e-llm-inference-service] To make an async_req request, set the async_req parameter. [e2e-llm-inference-service] [e2e-llm-inference-service] :param resource_path: Path to method endpoint. [e2e-llm-inference-service] :param method: Method to call. [e2e-llm-inference-service] :param path_params: Path parameters in the url. [e2e-llm-inference-service] :param query_params: Query parameters in the url. [e2e-llm-inference-service] :param header_params: Header parameters to be [e2e-llm-inference-service] placed in the request header. [e2e-llm-inference-service] :param body: Request body. [e2e-llm-inference-service] :param post_params dict: Request post form parameters, [e2e-llm-inference-service] for `application/x-www-form-urlencoded`, `multipart/form-data`. [e2e-llm-inference-service] :param auth_settings list: Auth Settings names for the request. [e2e-llm-inference-service] :param response: Response data type. [e2e-llm-inference-service] :param files dict: key -> filename, value -> filepath, [e2e-llm-inference-service] for `multipart/form-data`. [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param collection_formats: dict of collection formats for path, query, [e2e-llm-inference-service] header, and post parameters. [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: [e2e-llm-inference-service] If async_req parameter is True, [e2e-llm-inference-service] the request will be called asynchronously. [e2e-llm-inference-service] The method will return the request thread. [e2e-llm-inference-service] If parameter async_req is False or missing, [e2e-llm-inference-service] then the method will return the response directly. [e2e-llm-inference-service] """ [e2e-llm-inference-service] if not async_req: [e2e-llm-inference-service] > return self.__call_api(resource_path, method, [e2e-llm-inference-service] path_params, query_params, header_params, [e2e-llm-inference-service] body, post_params, files, [e2e-llm-inference-service] response_type, auth_settings, [e2e-llm-inference-service] _return_http_data_only, collection_formats, [e2e-llm-inference-service] _preload_content, _request_timeout, _host) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:348: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-inference-service-fac1c4ae/llminferenceservices/llmisvc-model-fb-opt-125m-route-3c960099' [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] path_params = [('group', 'serving.kserve.io'), ('version', 'v1alpha1'), ('namespace', 'e2e-test-llm-inference-service-fac1c4ae'), ('plural', 'llminferenceservices'), ('name', 'llmisvc-model-fb-opt-125m-route-3c960099')] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def __call_api( [e2e-llm-inference-service] self, resource_path, method, path_params=None, [e2e-llm-inference-service] query_params=None, header_params=None, body=None, post_params=None, [e2e-llm-inference-service] files=None, response_type=None, auth_settings=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] [e2e-llm-inference-service] config = self.configuration [e2e-llm-inference-service] [e2e-llm-inference-service] # header parameters [e2e-llm-inference-service] header_params = header_params or {} [e2e-llm-inference-service] header_params.update(self.default_headers) [e2e-llm-inference-service] if self.cookie: [e2e-llm-inference-service] header_params['Cookie'] = self.cookie [e2e-llm-inference-service] if header_params: [e2e-llm-inference-service] header_params = self.sanitize_for_serialization(header_params) [e2e-llm-inference-service] header_params = dict(self.parameters_to_tuples(header_params, [e2e-llm-inference-service] collection_formats)) [e2e-llm-inference-service] [e2e-llm-inference-service] # path parameters [e2e-llm-inference-service] if path_params: [e2e-llm-inference-service] path_params = self.sanitize_for_serialization(path_params) [e2e-llm-inference-service] path_params = self.parameters_to_tuples(path_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] for k, v in path_params: [e2e-llm-inference-service] # specified safe chars, encode everything [e2e-llm-inference-service] resource_path = resource_path.replace( [e2e-llm-inference-service] '{%s}' % k, [e2e-llm-inference-service] quote(str(v), safe=config.safe_chars_for_path_param) [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # query parameters [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] query_params = self.sanitize_for_serialization(query_params) [e2e-llm-inference-service] query_params = self.parameters_to_tuples(query_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] # post parameters [e2e-llm-inference-service] if post_params or files: [e2e-llm-inference-service] post_params = post_params if post_params else [] [e2e-llm-inference-service] post_params = self.sanitize_for_serialization(post_params) [e2e-llm-inference-service] post_params = self.parameters_to_tuples(post_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] post_params.extend(self.files_parameters(files)) [e2e-llm-inference-service] [e2e-llm-inference-service] # auth setting [e2e-llm-inference-service] self.update_params_for_auth(header_params, query_params, auth_settings) [e2e-llm-inference-service] [e2e-llm-inference-service] # body [e2e-llm-inference-service] if body: [e2e-llm-inference-service] body = self.sanitize_for_serialization(body) [e2e-llm-inference-service] [e2e-llm-inference-service] # request url [e2e-llm-inference-service] if _host is None: [e2e-llm-inference-service] url = self.configuration.host + resource_path [e2e-llm-inference-service] else: [e2e-llm-inference-service] # use server/host defined in path or operation instead [e2e-llm-inference-service] url = _host + resource_path [e2e-llm-inference-service] [e2e-llm-inference-service] # perform request and return response [e2e-llm-inference-service] > response_data = self.request( [e2e-llm-inference-service] method, url, query_params=query_params, headers=header_params, [e2e-llm-inference-service] post_params=post_params, body=body, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:180: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] url = 'https://afb35e615998e4024955f70c61cd36d7-c1814657712f161b.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-inference-service-fac1c4ae/llminferenceservices/llmisvc-model-fb-opt-125m-route-3c960099' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] post_params = [], body = None, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] post_params=None, body=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Makes the HTTP request using RESTClient.""" [e2e-llm-inference-service] if method == "GET": [e2e-llm-inference-service] > return self.rest_client.GET(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:373: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] url = 'https://afb35e615998e4024955f70c61cd36d7-c1814657712f161b.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-inference-service-fac1c4ae/llminferenceservices/llmisvc-model-fb-opt-125m-route-3c960099' [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] query_params = [], _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def GET(self, url, headers=None, query_params=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] > return self.request("GET", url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] query_params=query_params) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:244: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] url = 'https://afb35e615998e4024955f70c61cd36d7-c1814657712f161b.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-inference-service-fac1c4ae/llminferenceservices/llmisvc-model-fb-opt-125m-route-3c960099' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] body=None, post_params=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Perform requests. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: http request method [e2e-llm-inference-service] :param url: http request url [e2e-llm-inference-service] :param query_params: query parameters in the url [e2e-llm-inference-service] :param headers: http request headers [e2e-llm-inference-service] :param body: request json body, for `application/json` [e2e-llm-inference-service] :param post_params: request post parameters, [e2e-llm-inference-service] `application/x-www-form-urlencoded` [e2e-llm-inference-service] and `multipart/form-data` [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] """ [e2e-llm-inference-service] method = method.upper() [e2e-llm-inference-service] assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT', [e2e-llm-inference-service] 'PATCH', 'OPTIONS'] [e2e-llm-inference-service] [e2e-llm-inference-service] if post_params and body: [e2e-llm-inference-service] raise ApiValueError( [e2e-llm-inference-service] "body parameter cannot be used with post_params parameter." [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] post_params = post_params or {} [e2e-llm-inference-service] headers = headers or {} [e2e-llm-inference-service] [e2e-llm-inference-service] timeout = None [e2e-llm-inference-service] if _request_timeout: [e2e-llm-inference-service] if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)): # noqa: E501,F821 [e2e-llm-inference-service] timeout = urllib3.Timeout(total=_request_timeout) [e2e-llm-inference-service] elif (isinstance(_request_timeout, tuple) and [e2e-llm-inference-service] len(_request_timeout) == 2): [e2e-llm-inference-service] timeout = urllib3.Timeout( [e2e-llm-inference-service] connect=_request_timeout[0], read=_request_timeout[1]) [e2e-llm-inference-service] [e2e-llm-inference-service] if 'Content-Type' not in headers: [e2e-llm-inference-service] headers['Content-Type'] = 'application/json' [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE` [e2e-llm-inference-service] if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']: [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] url += '?' + urlencode(query_params) [e2e-llm-inference-service] if (re.search('json', headers['Content-Type'], re.IGNORECASE) or [e2e-llm-inference-service] headers['Content-Type'] == 'application/apply-patch+yaml'): [e2e-llm-inference-service] if headers['Content-Type'] == 'application/json-patch+json': [e2e-llm-inference-service] if not isinstance(body, list): [e2e-llm-inference-service] headers['Content-Type'] = \ [e2e-llm-inference-service] 'application/strategic-merge-patch+json' [e2e-llm-inference-service] request_body = None [e2e-llm-inference-service] if body is not None: [e2e-llm-inference-service] request_body = json.dumps(body) [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'application/x-www-form-urlencoded': # noqa: E501 [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=False, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'multipart/form-data': [e2e-llm-inference-service] # must del headers['Content-Type'], or the correct [e2e-llm-inference-service] # Content-Type which generated by urllib3 will be [e2e-llm-inference-service] # overwritten. [e2e-llm-inference-service] del headers['Content-Type'] [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=True, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] # Pass a `string` parameter directly in the body to support [e2e-llm-inference-service] # other content types than Json when `body` argument is [e2e-llm-inference-service] # provided in serialized form [e2e-llm-inference-service] elif isinstance(body, str) or isinstance(body, bytes): [e2e-llm-inference-service] request_body = body [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] else: [e2e-llm-inference-service] # Cannot generate the request from given parameters [e2e-llm-inference-service] msg = """Cannot prepare a request message for provided [e2e-llm-inference-service] arguments. Please check that your arguments match [e2e-llm-inference-service] declared content type.""" [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] # For `GET`, `HEAD` [e2e-llm-inference-service] else: [e2e-llm-inference-service] r = self.pool_manager.request(method, url, [e2e-llm-inference-service] fields=query_params, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] except urllib3.exceptions.SSLError as e: [e2e-llm-inference-service] msg = "{0}\n{1}".format(type(e).__name__, str(e)) [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] [e2e-llm-inference-service] if _preload_content: [e2e-llm-inference-service] r = RESTResponse(r) [e2e-llm-inference-service] [e2e-llm-inference-service] # In the python 3, the response.data is bytes. [e2e-llm-inference-service] # we need to decode it to string. [e2e-llm-inference-service] if six.PY3: [e2e-llm-inference-service] r.data = r.data.decode('utf8') [e2e-llm-inference-service] [e2e-llm-inference-service] # log response body [e2e-llm-inference-service] logger.debug("response body: %s", r.data) [e2e-llm-inference-service] [e2e-llm-inference-service] if not 200 <= r.status <= 299: [e2e-llm-inference-service] > raise ApiException(http_resp=r) [e2e-llm-inference-service] E kubernetes.client.exceptions.ApiException: (500) [e2e-llm-inference-service] E Reason: Internal Server Error [e2e-llm-inference-service] E HTTP response headers: HTTPHeaderDict({'Audit-Id': '79b38d05-672e-4258-820a-2ec2a1b528f5', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'bf546a5c-3f60-4261-9b3f-c3b9e58fb8c8', 'X-Kubernetes-Pf-Prioritylevel-Uid': '39b8073e-7e81-4403-a6ca-a29718cbfcdd', 'Date': 'Mon, 27 Jul 2026 17:15:41 GMT', 'Content-Length': '264'}) [e2e-llm-inference-service] E HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"conversion webhook for serving.kserve.io/v1alpha2, Kind=LLMInferenceService failed: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/convert?timeout=30s\": EOF","code":500} [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:238: ApiException [e2e-llm-inference-service] [e2e-llm-inference-service] The above exception was the direct cause of the following exception: [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-no-scheduler', 'workload-single-cpu', 'model-fb-opt-125m'], prompt='What is KServe?', serv... {'name': 'model-fb-opt-125m-llmisvc-model-7a2ca70d'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway="router-gateway-2", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-3", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-4", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] after_test=[ [e2e-llm-inference-service] lambda tc: delete_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — clean path: token-producer in inline config [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-tokenizer-kvcache", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-clean-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — migration path: legacy precise-prefix-cache-scorer [e2e-llm-inference-service] # triggers auto-provisioned tokenizer without explicit tokenizer:{} field [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-migration-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name="publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:866: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...c-mod-5c1136e5'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-llmisvc-model-7a2ca70d'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-27T17:13:40.737483', start_time = 1785172420.7380967 [e2e-llm-inference-service] duration = 120.4102156162262, timestamp_end = '2026-07-27T17:15:41.148319' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....llmisvc-mod-5c1136e5'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-llmisvc-model-7a2ca70d'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f8e575ec9a0> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] > out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1339: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] name = 'llmisvc-model-fb-opt-125m-route-3c960099' [e2e-llm-inference-service] namespace = 'e2e-test-llm-inference-service-fac1c4ae', version = 'v1alpha1' [e2e-llm-inference-service] [e2e-llm-inference-service] def get_llmisvc( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] name, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] version=constants.KSERVE_V1ALPHA1_VERSION, [e2e-llm-inference-service] ): [e2e-llm-inference-service] try: [e2e-llm-inference-service] return kserve_client.api_instance.get_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICE, [e2e-llm-inference-service] name, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except client.rest.ApiException as e: [e2e-llm-inference-service] > raise RuntimeError( [e2e-llm-inference-service] f"❌ Exception when calling CustomObjectsApi->" [e2e-llm-inference-service] f"get_namespaced_custom_object for LLMInferenceService: {e}" [e2e-llm-inference-service] ) from e [e2e-llm-inference-service] E RuntimeError: ❌ Exception when calling CustomObjectsApi->get_namespaced_custom_object for LLMInferenceService: (500) [e2e-llm-inference-service] E Reason: Internal Server Error [e2e-llm-inference-service] E HTTP response headers: HTTPHeaderDict({'Audit-Id': '79b38d05-672e-4258-820a-2ec2a1b528f5', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'bf546a5c-3f60-4261-9b3f-c3b9e58fb8c8', 'X-Kubernetes-Pf-Prioritylevel-Uid': '39b8073e-7e81-4403-a6ca-a29718cbfcdd', 'Date': 'Mon, 27 Jul 2026 17:15:41 GMT', 'Content-Length': '264'}) [e2e-llm-inference-service] E HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"conversion webhook for serving.kserve.io/v1alpha2, Kind=LLMInferenceService failed: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/convert?timeout=30s\": EOF","code":500} [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1212: RuntimeError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:488 Gateway diagnostics started (interval=300s, cli=oc, output=/workspace/artifacts-dir/gateway_diagnostics.jsonl) [e2e-llm-inference-service] INFO kserve:conftest.py:40 Logger configured [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-inference-service-fac1c4ae with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-no-scheduler-llmisvc-mod-ff50d279 in namespace e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-no-scheduler-llmisvc-mod-ff50d279 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-no-scheduler-llmisvc-mod-ff50d279 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-single-cpu-llmisvc-mod-5c1136e5 in namespace e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-single-cpu-llmisvc-mod-5c1136e5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-single-cpu-llmisvc-mod-5c1136e5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig model-fb-opt-125m-llmisvc-model-7a2ca70d in namespace e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig model-fb-opt-125m-llmisvc-model-7a2ca70d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig model-fb-opt-125m-llmisvc-model-7a2ca70d [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:82 Updated ConfigMap kserve-ci-e2e-test/gateway-proxy-config [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-07-27T17:13:40.586965] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-no-scheduler', 'workload-single-cpu', 'model-fb-opt-125m'], prompt='What is KServe?', service_name='llmisvc-model-fb-opt-125m-route-3c960099', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-inference-service-fac1c4ae', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-route-3c960099', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-fac1c4ae', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-no-scheduler-llmisvc-mod-ff50d279'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-5c1136e5'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-llmisvc-model-7a2ca70d'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-27T17:13:40.601832] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-route-3c960099', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-fac1c4ae', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-no-scheduler-llmisvc-mod-ff50d279'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-5c1136e5'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-llmisvc-model-7a2ca70d'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-27T17:13:40.737268] end - ✅ in 0.135s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-27T17:13:40.737483] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-route-3c960099', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-fac1c4ae', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-no-scheduler-llmisvc-mod-ff50d279'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-5c1136e5'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-llmisvc-model-7a2ca70d'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 0: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "0\t{\"running\":{\"startedAt\":\"2026-07-27T17:09:22Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-27T17:14:11Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route: v1.HTTPRouteStatus{RouteStatus:v1.RouteStatus{Parents:[]v1.RouteParentStatus(nil)}}]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:14:11Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:14:11Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:14:11Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route: v1.HTTPRouteStatus{RouteStatus:v1.RouteStatus{Parents:[]v1.RouteParentStatus(nil)}}]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:14:11Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route: v1.HTTPRouteStatus{RouteStatus:v1.RouteStatus{Parents:[]v1.RouteParentStatus(nil)}}]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:14:11Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-27T17:14:18Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:14:18Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:14:11Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:14:18Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:14:18Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:14:18Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-27T17:15:41.148319] end - ❌ 120.410s: ❌ Exception when calling CustomObjectsApi->get_namespaced_custom_object for LLMInferenceService: (500) [e2e-llm-inference-service] Reason: Internal Server Error [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '79b38d05-672e-4258-820a-2ec2a1b528f5', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'bf546a5c-3f60-4261-9b3f-c3b9e58fb8c8', 'X-Kubernetes-Pf-Prioritylevel-Uid': '39b8073e-7e81-4403-a6ca-a29718cbfcdd', 'Date': 'Mon, 27 Jul 2026 17:15:41 GMT', 'Content-Length': '264'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"conversion webhook for serving.kserve.io/v1alpha2, Kind=LLMInferenceService failed: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/convert?timeout=30s\": EOF","code":500} [e2e-llm-inference-service] [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:903 [router-no-scheduler-workload-single-cpu-model-fb-opt-125m] ❌ ERROR: Failed to call llm inference service llmisvc-model-fb-opt-125m-route-3c960099: ❌ Exception when calling CustomObjectsApi->get_namespaced_custom_object for LLMInferenceService: (500) [e2e-llm-inference-service] Reason: Internal Server Error [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '79b38d05-672e-4258-820a-2ec2a1b528f5', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'bf546a5c-3f60-4261-9b3f-c3b9e58fb8c8', 'X-Kubernetes-Pf-Prioritylevel-Uid': '39b8073e-7e81-4403-a6ca-a29718cbfcdd', 'Date': 'Mon, 27 Jul 2026 17:15:41 GMT', 'Content-Length': '264'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"conversion webhook for serving.kserve.io/v1alpha2, Kind=LLMInferenceService failed: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/convert?timeout=30s\": EOF","code":500} [e2e-llm-inference-service] [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:243 # Diagnostics for 'llmisvc-model-fb-opt-125m-route-3c960099' in 'e2e-test-llm-inference-service-fac1c4ae' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:244 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:247 # LLMInferenceService llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:258 # failed to dump LLMInferenceService: (500) [e2e-llm-inference-service] Reason: Internal Server Error [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '27515e32-e28e-428c-8ead-460e831d83c3', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'bf546a5c-3f60-4261-9b3f-c3b9e58fb8c8', 'X-Kubernetes-Pf-Prioritylevel-Uid': '39b8073e-7e81-4403-a6ca-a29718cbfcdd', 'Date': 'Mon, 27 Jul 2026 17:15:41 GMT', 'Content-Length': '264'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"conversion webhook for serving.kserve.io/v1alpha2, Kind=LLMInferenceService failed: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/convert?timeout=30s\": EOF","code":500} [e2e-llm-inference-service] [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:68 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:69 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:14:10 e2e-test-llm-inference-service-fac1c4ae replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-fac1c4ae Normal Scheduled Successfully assigned e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4 to ip-10-0-143-86.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:14:10 e2e-test-llm-inference-service-fac1c4ae multus Normal AddedInterface Add eth0 [10.132.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:14:10 e2e-test-llm-inference-service-fac1c4ae kubelet/ip-10-0-143-86.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:766bf293cdc0c4d8490474470e0c0992ef88060d1540c60d45ae6827b1115e1c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:14:11 e2e-test-llm-inference-service-fac1c4ae kubelet/ip-10-0-143-86.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:14:11 e2e-test-llm-inference-service-fac1c4ae kubelet/ip-10-0-143-86.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:14:19 e2e-test-llm-inference-service-fac1c4ae kubelet/ip-10-0-143-86.ec2.internal Normal Pulling Pulling image "vllm/vllm-openai-cpu:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:14:49 e2e-test-llm-inference-service-fac1c4ae kubelet/ip-10-0-143-86.ec2.internal Normal Pulled Successfully pulled image "vllm/vllm-openai-cpu:v0.19.0" in 30.206s (30.206s including waiting). Image size: 3531178100 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:14:49 e2e-test-llm-inference-service-fac1c4ae kubelet/ip-10-0-143-86.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:14:49 e2e-test-llm-inference-service-fac1c4ae kubelet/ip-10-0-143-86.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:15:40 e2e-test-llm-inference-service-fac1c4ae kubelet/ip-10-0-143-86.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.44:8000/health": dial tcp 10.132.0.44:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:14:10 e2e-test-llm-inference-service-fac1c4ae deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:14:01 e2e-test-llm-inference-service-fac1c4ae OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-3c960099-kserve-route-authn: failed to get HTTPRoute e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-3c960099-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:14:10 e2e-test-llm-inference-service-fac1c4ae LLMInferenceServiceController Normal Created Created v1.Secret e2e-test-llm-inference-service-fac1c4ae/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:14:10 e2e-test-llm-inference-service-fac1c4ae LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:14:10 e2e-test-llm-inference-service-fac1c4ae LLMInferenceServiceController Normal Created Created v1.Service e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:14:10 e2e-test-llm-inference-service-fac1c4ae LLMInferenceServiceController Normal Created Created v1.HTTPRoute e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:14:10 e2e-test-llm-inference-service-fac1c4ae LLMInferenceServiceController Normal Created Created v1.DestinationRule e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:14:18 e2e-test-llm-inference-service-fac1c4ae LLMInferenceServiceController Normal Updated Updated v1.Secret e2e-test-llm-inference-service-fac1c4ae/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4 (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### init-container 'storage-initializer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:228 # -- logs (current): unavailable ((500) [e2e-llm-inference-service] Reason: Internal Server Error [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '8aa1d414-ce21-4b68-83a0-d50a2b53e9b7', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Mon, 27 Jul 2026 17:15:41 GMT', 'Content-Length': '283'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Get \"https://10.0.143.86:10250/containerLogs/e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4/storage-initializer?tailLines=200\": EOF","code":500} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:228 # -- logs (current): unavailable ((500) [e2e-llm-inference-service] Reason: Internal Server Error [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': 'e703c5be-9a29-4d06-b201-2bad5920b056', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Mon, 27 Jul 2026 17:15:41 GMT', 'Content-Length': '268'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Get \"https://10.0.143.86:10250/containerLogs/e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4/main?tailLines=200\": EOF","code":500} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] uid: 6bfb867f-53d2-4f89-88cb-b624535f6c32 [e2e-llm-inference-service] resourceVersion: '22250' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T17:14:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - notReadyAddresses: [e2e-llm-inference-service] - ip: 10.132.0.44 [e2e-llm-inference-service] nodeName: ip-10-0-143-86.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4 [e2e-llm-inference-service] uid: 23577b52-1aec-4f44-af44-29801987ec61 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4 [e2e-llm-inference-service] generateName: llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c5- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] uid: 23577b52-1aec-4f44-af44-29801987ec61 [e2e-llm-inference-service] resourceVersion: '22857' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 59dc67d6c5 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.44/23"],"mac_address":"0a:58:0a:84:00:2c","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.44/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.132.0.44\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:84:00:2c\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c5 [e2e-llm-inference-service] uid: 2c6b6760-9b68-411f-85f2-516c9e392d83 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-143-86 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2c6b6760-9b68-411f-85f2-516c9e392d83"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TOKIO_WORKER_THREADS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T17:14:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.132.0.44"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kube-api-access-qjvzb [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:766bf293cdc0c4d8490474470e0c0992ef88060d1540c60d45ae6827b1115e1c [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKIO_WORKER_THREADS [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-qjvzb [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000730000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+\ [e2e-llm-inference-service] \ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" != *\"\ [e2e-llm-inference-service] --kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"* ]] &&\ [e2e-llm-inference-service] \ [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\"\ [e2e-llm-inference-service] \n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"\ [e2e-llm-inference-service] facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-qjvzb [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000730000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-143-86.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c27,c14 [e2e-llm-inference-service] fsGroup: 1000730000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-np22w [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T17:14:11Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T17:14:19Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] hostIP: 10.0.143.86 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.143.86 [e2e-llm-inference-service] podIP: 10.132.0.44 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.132.0.44 [e2e-llm-inference-service] startTime: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] state: [e2e-llm-inference-service] terminated: [e2e-llm-inference-service] exitCode: 0 [e2e-llm-inference-service] reason: Completed [e2e-llm-inference-service] startedAt: '2026-07-27T17:14:11Z' [e2e-llm-inference-service] finishedAt: '2026-07-27T17:14:18Z' [e2e-llm-inference-service] containerID: cri-o://d8b45ad26d0472523c21b3972af4cb01d1ede61de310c0cb08d69dba94e495cb [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:766bf293cdc0c4d8490474470e0c0992ef88060d1540c60d45ae6827b1115e1c [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-storage-initializer@sha256:520104f58d6aa1e9aa07e7a060ffac702804a8e4e371d8bf953126c0464a8ee1 [e2e-llm-inference-service] containerID: cri-o://d8b45ad26d0472523c21b3972af4cb01d1ede61de310c0cb08d69dba94e495cb [e2e-llm-inference-service] started: false [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-qjvzb [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000730000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000730000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-27T17:14:49Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: docker.io/vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: docker.io/vllm/vllm-openai-cpu@sha256:a8257c201fd2f696c146615987be61579aa2052367b73c27bd3fb1ab8e90bd4b [e2e-llm-inference-service] containerID: cri-o://7e9ac095b7b6bbe8b475897af48f45056a66cf764eff352c2b83503723c40474 [e2e-llm-inference-service] started: false [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-qjvzb [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000730000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000730000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] uid: d04ccf16-06ba-4e03-ab5f-bfd4abbd64e1 [e2e-llm-inference-service] resourceVersion: '22150' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] uid: 50286ce7-e4b5-4a90-b8b6-7ce820135097 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"50286ce7-e4b5-4a90-b8b6-7ce820135097"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.40.192 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.40.192 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] uid: f7790a95-f47a-4fde-b0c5-ba1bb5c958fb [e2e-llm-inference-service] resourceVersion: '22170' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] uid: 50286ce7-e4b5-4a90-b8b6-7ce820135097 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:unavailableReplicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"50286ce7-e4b5-4a90-b8b6-7ce820135097"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TOKIO_WORKER_THREADS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:766bf293cdc0c4d8490474470e0c0992ef88060d1540c60d45ae6827b1115e1c [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKIO_WORKER_THREADS [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] unavailableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] reason: ReplicaSetUpdated [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c5" [e2e-llm-inference-service] is progressing. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c5 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] uid: 2c6b6760-9b68-411f-85f2-516c9e392d83 [e2e-llm-inference-service] resourceVersion: '22165' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 59dc67d6c5 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve [e2e-llm-inference-service] uid: f7790a95-f47a-4fde-b0c5-ba1bb5c958fb [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"f7790a95-f47a-4fde-b0c5-ba1bb5c958fb"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TOKIO_WORKER_THREADS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 59dc67d6c5 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 59dc67d6c5 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:766bf293cdc0c4d8490474470e0c0992ef88060d1540c60d45ae6827b1115e1c [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKIO_WORKER_THREADS [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-sdqdst [e2e-llm-inference-service] generateName: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] uid: 21f818b6-1207-4c5a-bca2-aa2d78cd7bb1 [e2e-llm-inference-service] resourceVersion: '22249' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] uid: d04ccf16-06ba-4e03-ab5f-bfd4abbd64e1 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-27T17:14:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d04ccf16-06ba-4e03-ab5f-bfd4abbd64e1"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.132.0.44 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] serving: false [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4 [e2e-llm-inference-service] uid: 23577b52-1aec-4f44-af44-29801987ec61 [e2e-llm-inference-service] nodeName: ip-10-0-143-86.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"50286ce7-e4b5-4a90-b8b6-7ce820135097"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-27T17:14:12Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] uid: 50286ce7-e4b5-4a90-b8b6-7ce820135097 [e2e-llm-inference-service] resourceVersion: '22301' [e2e-llm-inference-service] uid: 3fb1f0a4-f238-4e0c-a09c-7d7d3b15f803 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T17:14:11Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"50286ce7-e4b5-4a90-b8b6-7ce820135097"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-27T17:14:12Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] uid: 50286ce7-e4b5-4a90-b8b6-7ce820135097 [e2e-llm-inference-service] resourceVersion: '22301' [e2e-llm-inference-service] uid: 3fb1f0a4-f238-4e0c-a09c-7d7d3b15f803 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-fac1c4ae/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T17:14:11Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:14:11Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"50286ce7-e4b5-4a90-b8b6-7ce820135097"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T17:14:11Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-27T17:14:13Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-route-authn [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] uid: 50286ce7-e4b5-4a90-b8b6-7ce820135097 [e2e-llm-inference-service] resourceVersion: '22323' [e2e-llm-inference-service] uid: 36111aa5-baad-4af4-92a8-9d5d4fccb978 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T17:14:12Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T17:14:13Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"50286ce7-e4b5-4a90-b8b6-7ce820135097"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] uid: 50286ce7-e4b5-4a90-b8b6-7ce820135097 [e2e-llm-inference-service] resourceVersion: '22163' [e2e-llm-inference-service] uid: c36c2ff7-c67f-4489-922c-4d3cac7aa034 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc.e2e-test-llm-inference-service-fac1c4ae.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc.e2e-test-llm-inference-service-fac1c4ae.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"50286ce7-e4b5-4a90-b8b6-7ce820135097"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] uid: 50286ce7-e4b5-4a90-b8b6-7ce820135097 [e2e-llm-inference-service] resourceVersion: '22163' [e2e-llm-inference-service] uid: c36c2ff7-c67f-4489-922c-4d3cac7aa034 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc.e2e-test-llm-inference-service-fac1c4ae.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc.e2e-test-llm-inference-service-fac1c4ae.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"50286ce7-e4b5-4a90-b8b6-7ce820135097"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T17:14:10Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] uid: 50286ce7-e4b5-4a90-b8b6-7ce820135097 [e2e-llm-inference-service] resourceVersion: '22163' [e2e-llm-inference-service] uid: c36c2ff7-c67f-4489-922c-4d3cac7aa034 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc.e2e-test-llm-inference-service-fac1c4ae.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc.e2e-test-llm-inference-service-fac1c4ae.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-fac1c4ae [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:15:42Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-3c960099 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 59dc67d6c5 [e2e-llm-inference-service] timestamp: '2026-07-27T17:15:16Z' [e2e-llm-inference-service] window: 15.851s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 1015668285n [e2e-llm-inference-service] memory: 258344Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-07-27T17:15:42.466988] end - ❌ 121.880s: ❌ Exception when calling CustomObjectsApi->get_namespaced_custom_object for LLMInferenceService: (500) [e2e-llm-inference-service] Reason: Internal Server Error [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '79b38d05-672e-4258-820a-2ec2a1b528f5', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'bf546a5c-3f60-4261-9b3f-c3b9e58fb8c8', 'X-Kubernetes-Pf-Prioritylevel-Uid': '39b8073e-7e81-4403-a6ca-a29718cbfcdd', 'Date': 'Mon, 27 Jul 2026 17:15:41 GMT', 'Content-Length': '264'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"conversion webhook for serving.kserve.io/v1alpha2, Kind=LLMInferenceService failed: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/convert?timeout=30s\": EOF","code":500} [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-inference-service-fac1c4ae (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_hpa_deployment[router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_hpa [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-no-replicas", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-hpa", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-hpa-deploy", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_hpa_deployment(test_case: TestCase): [e2e-llm-inference-service] """HPA + Deployment: HPA exists with WVA annotations; pods scale up under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:542: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-27T17:25:59.528654', start_time = 1785173159.5290842 [e2e-llm-inference-service] duration = 900.2886075973511, timestamp_end = '2026-07-27T17:40:59.817712' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f4e06fc9300> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:26:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:26:39Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:26:42Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:26:30Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:27:01Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:27:01Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:26:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:27:01Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:26:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_hpa_deployment] [2026-07-27T17:25:59.058599] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prompt='KServe is a', service_name='autoscale-hpa-deploy', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-27T17:25:59.072757] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-27T17:25:59.528457] end - ✅ in 0.455s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-27T17:25:59.528654] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:26:30Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:26:30Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:26:30Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:26:30Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:26:30Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:26:30Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:26:30Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:26:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:26:30Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:26:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:26:39Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:26:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:26:30Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:26:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:26:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:26:30Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:26:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:26:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:26:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:26:39Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:26:42Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:26:30Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:26:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:26:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:26:30Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:26:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:26:42Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:26:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:26:39Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:26:42Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:26:30Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:26:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:26:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:26:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:26:30Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:26:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:26:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:26:39Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:26:42Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:26:30Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:27:01Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:27:01Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:26:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:27:01Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:26:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:26:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:26:39Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:26:42Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:26:30Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:27:01Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:27:01Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:26:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:27:01Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:26:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-27T17:40:59.817712] end - ❌ 900.289s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:26:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:26:39Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:26:42Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:26:30Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:27:01Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:27:01Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:26:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:27:01Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:26:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-27T17:40:59.817977] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 2 pod(s) for autoscale-hpa-deploy still terminating: ['autoscale-hpa-deploy-kserve-66746bd8b9-crcfs', 'autoscale-hpa-deploy-kserve-router-scheduler-94df769b6-d6x2d'] [e2e-llm-inference-service] assert not ['autoscale-hpa-deploy-kserve-66746bd8b9-crcfs', 'autoscale-hpa-deploy-kserve-router-scheduler-94df769b6-d6x2d'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-hpa-deploy still terminating: ['autoscale-hpa-deploy-kserve-router-scheduler-94df769b6-d6x2d'] [e2e-llm-inference-service] assert not ['autoscale-hpa-deploy-kserve-router-scheduler-94df769b6-d6x2d'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-27T17:41:19.936925] end - ✅ in 20.118s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_hpa_deployment] [2026-07-27T17:41:19.937094] end - ❌ 920.878s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:26:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:26:39Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:26:42Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:26:30Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:27:01Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:27:01Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:26:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:27:01Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:26:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_keda_deployment[router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-keda'], pro... {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_keda [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-no-replicas", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-keda", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-keda-deploy", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_keda_deployment(test_case: TestCase): [e2e-llm-inference-service] """KEDA + Deployment: ScaledObject exists with WVA annotations; no HPA; pods scale up under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:606: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-keda'], pro... {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-27T17:41:20.554674', start_time = 1785174080.555023 [e2e-llm-inference-service] duration = 900.2079703807831, timestamp_end = '2026-07-27T17:56:20.762996' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f4e06fc98a0> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:41:54Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:41:54Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:41:54Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:41:41Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:42:12Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:42:12Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:41:54Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:42:12Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:41:54Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-keda-deployment-b2150d0b with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-keda-d-27e06c40 in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-keda-d-27e06c40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-keda-d-27e06c40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-no-repl-da49e827 in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-no-repl-da49e827 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-no-repl-da49e827 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-101f2a9d in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-101f2a9d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-101f2a9d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-keda-autoscale-keda-dep-1ac84077 in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-keda-autoscale-keda-dep-1ac84077 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-keda-autoscale-keda-dep-1ac84077 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_keda_deployment] [2026-07-27T17:41:20.475847] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-keda'], prompt='KServe is a', service_name='autoscale-keda-deploy', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-keda-deployment-b2150d0b', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-27T17:41:20.488761] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-27T17:41:20.554502] end - ✅ in 0.065s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-27T17:41:20.554674] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:41:41Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:41:41Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:41:41Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:41:41Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:41:41Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:41:41Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:41:41Z', 'message': 'ScaledObject not yet visible in cache', 'reason': 'ScaledObjectProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:41:41Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:41:41Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:41:54Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:41:54Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:41:54Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:41:41Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:41:54Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:41:54Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:41:54Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:41:54Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:41:54Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:41:54Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:41:54Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:41:54Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:41:41Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:42:12Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:42:12Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:41:54Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:42:12Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:41:54Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:41:54Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:41:54Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:41:54Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:41:41Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:42:12Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:42:12Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:41:54Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:42:12Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:41:54Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-27T17:56:20.762996] end - ❌ 900.208s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:41:54Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:41:54Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:41:54Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:41:41Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:42:12Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:42:12Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:41:54Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:42:12Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:41:54Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-27T17:56:20.763085] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 2 pod(s) for autoscale-keda-deploy still terminating: ['autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95', 'autoscale-keda-deploy-kserve-router-scheduler-6499b6dbcf-r59fd'] [e2e-llm-inference-service] assert not ['autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95', 'autoscale-keda-deploy-kserve-router-scheduler-6499b6dbcf-r59fd'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-keda-deploy still terminating: ['autoscale-keda-deploy-kserve-router-scheduler-6499b6dbcf-r59fd'] [e2e-llm-inference-service] assert not ['autoscale-keda-deploy-kserve-router-scheduler-6499b6dbcf-r59fd'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-27T17:56:46.261132] end - ✅ in 25.498s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_keda_deployment] [2026-07-27T17:56:46.261369] end - ❌ 925.785s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:41:54Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:41:54Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:41:54Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:41:41Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:42:12Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:42:12Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:41:54Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:42:12Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:41:54Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _____ test_llm_inference_service[router-managed-workload-pd-cpu-model-pvc] _____ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-pd-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-mod... {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway="router-gateway-2", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-3", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-4", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] after_test=[ [e2e-llm-inference-service] lambda tc: delete_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — clean path: token-producer in inline config [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-tokenizer-kvcache", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-clean-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — migration path: legacy precise-prefix-cache-scorer [e2e-llm-inference-service] # triggers auto-provisioned tokenizer without explicit tokenizer:{} field [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-migration-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name="publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:866: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...del-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-27T17:45:52.438968', start_time = 1785174352.439295 [e2e-llm-inference-service] duration = 900.6989901065826, timestamp_end = '2026-07-27T18:00:53.138288' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....svc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f8e573d4680> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-27T17:46:28Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:48:35Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:46:17Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:46:54Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:46:54Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-inference-service-48639af5 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1959 Created PVC e2e-pvc-model-storage in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2099 Created model download Job e2e-pvc-model-download in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2124 Model download Job e2e-pvc-model-download completed successfully [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-07-27T17:45:52.291177] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-pd-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-model-pvc-router-manage-e8706282', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-inference-service-48639af5', before_test=[ at 0x7f8e5c6cbf60>], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-48639af5', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-27T17:45:52.303784] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-48639af5', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-27T17:45:52.438712] end - ✅ in 0.135s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-27T17:45:52.438968] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-48639af5', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-27T17:46:17Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:46:17Z', 'message': 'Inference Pool e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:46:17Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:46:17Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:46:17Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:46:17Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:46:17Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:46:17Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:46:17Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-27T17:46:28Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:46:17Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-27T17:46:28Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:46:17Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:46:54Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:46:54Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-27T17:46:28Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:48:35Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:46:17Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:46:54Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:46:54Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 7: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-27T17:14:03Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 8: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-27T17:14:03Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 9: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-27T17:14:03Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-27T17:46:28Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:48:35Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:46:17Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:46:54Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:46:54Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-27T18:00:53.138288] end - ❌ 900.699s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-27T17:46:28Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:48:35Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:46:17Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:46:54Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:46:54Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:903 [router-managed-workload-pd-cpu-model-pvc] ❌ ERROR: Failed to call llm inference service llmisvc-model-pvc-router-manage-e8706282: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-27T17:46:28Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:48:35Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:46:17Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:46:54Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:46:54Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:243 # Diagnostics for 'llmisvc-model-pvc-router-manage-e8706282' in 'e2e-test-llm-inference-service-48639af5' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:244 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:247 # LLMInferenceService llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:256 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:45:52Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T17:45:52Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T17:45:52Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler-latency-predictor: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tokenizer: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:prefill: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-27T17:48:35Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] resourceVersion: '59788' [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] - name: workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] - name: model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://a65b83380762d45cfa23aeddb3ae822c-1808012620.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a65b83380762d45cfa23aeddb3ae822c-1808012620.us-east-1.elb.amazonaws.com/e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a65b83380762d45cfa23aeddb3ae822c-1808012620.us-east-1.elb.amazonaws.com/publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler-latency-predictor: kserve-config-llm-scheduler-latency-predictor [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tokenizer: kserve-config-llm-tokenizer [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T17:46:28Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T17:46:28Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T17:48:35Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T17:46:28Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: PrefillWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T17:46:17Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T17:46:28Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T17:46:54Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T17:46:54Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T17:46:28Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://a65b83380762d45cfa23aeddb3ae822c-1808012620.us-east-1.elb.amazonaws.com/e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:68 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:69 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4255 to ip-10-0-143-86.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:45:37 e2e-test-llm-inference-service-48639af5 attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-56638b18-3f61-49c9-8c07-2721db06574b" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:45:39 e2e-test-llm-inference-service-48639af5 multus Normal AddedInterface Add eth0 [10.132.0.63/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:45:39 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-143-86.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:766bf293cdc0c4d8490474470e0c0992ef88060d1540c60d45ae6827b1115e1c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:45:39 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-143-86.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:45:39 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-143-86.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:45:32 e2e-test-llm-inference-service-48639af5 job-controller Normal SuccessfulCreate Created pod: e2e-pvc-model-download-m4255 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:45:51 e2e-test-llm-inference-service-48639af5 job-controller Normal Completed Job completed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:45:31 e2e-test-llm-inference-service-48639af5 persistentvolume-controller Normal WaitForFirstConsumer waiting for first consumer to be created before binding [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:45:32 e2e-test-llm-inference-service-48639af5 persistentvolume-controller Normal ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:45:32 e2e-test-llm-inference-service-48639af5 ebs.csi.aws.com_aws-ebs-csi-driver-controller-56688b566d-ccsf5_2334de51-c6fa-4ff5-8f28-b46a03a0b034 Normal Provisioning External provisioner is provisioning volume for claim "e2e-test-llm-inference-service-48639af5/e2e-pvc-model-storage" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:45:34 e2e-test-llm-inference-service-48639af5 ebs.csi.aws.com_aws-ebs-csi-driver-controller-56688b566d-ccsf5_2334de51-c6fa-4ff5-8f28-b46a03a0b034 Normal ProvisioningSucceeded Successfully provisioned volume pvc-56638b18-3f61-49c9-8c07-2721db06574b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:16 e2e-test-llm-inference-service-48639af5 replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582 to ip-10-0-143-86.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:18 e2e-test-llm-inference-service-48639af5 attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-56638b18-3f61-49c9-8c07-2721db06574b" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:20 e2e-test-llm-inference-service-48639af5 multus Normal AddedInterface Add eth0 [10.132.0.64/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:20 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-143-86.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:22 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-143-86.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0" in 2.419s (2.419s including waiting). Image size: 166803393 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:22 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-143-86.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:22 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-143-86.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:23 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-143-86.ec2.internal Normal Pulled Container image "vllm/vllm-openai-cpu:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:23 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-143-86.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:23 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-143-86.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:48:10 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-143-86.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.64:8001/health": dial tcp 10.132.0.64:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:16 e2e-test-llm-inference-service-48639af5 replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb to ip-10-0-132-194.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:18 e2e-test-llm-inference-service-48639af5 attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-56638b18-3f61-49c9-8c07-2721db06574b" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:16 e2e-test-llm-inference-service-48639af5 deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:15 e2e-test-llm-inference-service-48639af5 deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:12 e2e-test-llm-inference-service-48639af5 OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn: failed to get HTTPRoute e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-e8706282-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:15 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Secret e2e-test-llm-inference-service-48639af5/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:15 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:15 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Role e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:15 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.RoleBinding e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:15 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:15 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:16 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Service e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:16 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:16 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /e2e-test-llm-inference-service-132820f03254470cd8a0146008974a5f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:28 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:56:17 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Updated Updated v1.Secret e2e-test-llm-inference-service-48639af5/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:27 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:16 e2e-test-llm-inference-service-48639af5 replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:16 e2e-test-llm-inference-service-48639af5 deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-789b8c94b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7 to ip-10-0-132-194.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:16 e2e-test-llm-inference-service-48639af5 multus Normal AddedInterface Add eth0 [10.134.0.58/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:16 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-132-194.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:16 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-132-194.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-27 17:46:16 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-132-194.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582 (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### init-container 'llm-d-routing-sidecar' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 Flag --decoder-use-tls has been deprecated, use --enable-tls=decoder instead [e2e-llm-inference-service] Flag --prefiller-use-tls has been deprecated, use --enable-tls=prefiller instead [e2e-llm-inference-service] {"level":"info","ts":1785174382.905056,"msg":"Proxy starting","Built on":"","From Git SHA":"unknown"} [e2e-llm-inference-service] {"level":"info","ts":1785174382.9054017,"msg":"Proxy configuration","config":"{\"Port\":\"8000\",\"KVConnector\":\"nixlv2\",\"ECConnector\":\"\",\"DataParallelSize\":1,\"MaxIdleConnsPerHost\":1024,\"EnablePrefillerSampling\":false,\"PrefillMaxRetries\":0,\"PrefillRetryBackoff\":200000000,\"UseTLSForPrefiller\":true,\"UseTLSForDecoder\":true,\"UseTLSForEncoder\":false,\"InsecureSkipVerifyForPrefiller\":false,\"InsecureSkipVerifyForEncoder\":false,\"InsecureSkipVerifyForDecoder\":false,\"SecureServing\":true,\"CertPath\":\"/var/run/kserve/tls\",\"MooncakeBootstrapPort\":8998,\"EnableSSRFProtection\":true,\"InferencePoolNamespace\":\"e2e-test-llm-inference-service-48639af5\",\"InferencePoolName\":\"llmisvc-model-pvc-router-manage-e8706282-inference-pool\",\"PoolGroup\":\"inference.networking.x-k8s.io\",\"DecodeChunkSize\":0,\"Tracing\":false,\"MoRIIOWriteMode\":false,\"MoRIIODecodeNotifyPort\":61005,\"MoRIIODecodeHandshakePort\":6301,\"MoRIIODecodePodIP\":\"\",\"MoRIIOParallelDispatch\":false,\"MoRIIOPrefillHandshakePort\":6301,\"MoRIIOPrefillNotifyPort\":61005,\"MoRIIOTPSize\":1,\"MoRIIODPSize\":1,\"MoRIIORemoteHosts\":null,\"MoRIIODPSizeLocal\":0,\"MoRIIODecodeHosts\":null,\"DecoderURL\":\"https://localhost:8001\"}"} [e2e-llm-inference-service] {"level":"info","ts":1785174382.9060652,"logger":"allowlist-validator","msg":"starting SSRF protection allowlist validator","namespace":"e2e-test-llm-inference-service-48639af5","poolName":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","gvr":"inference.networking.x-k8s.io/v1alpha2, Resource=inferencepools"} [e2e-llm-inference-service] {"level":"info","ts":1785174382.9215713,"logger":"allowlist-validator","msg":"InferencePool added","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785174382.930398,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":0,"targets":{}} [e2e-llm-inference-service] {"level":"info","ts":1785174382.9304693,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":0,"targets":{}} [e2e-llm-inference-service] {"level":"info","ts":1785174383.0065722,"logger":"allowlist-validator","msg":"allowlist validator started successfully"} [e2e-llm-inference-service] {"level":"info","ts":1785174383.0086288,"logger":"proxy server on port 8000","msg":"server TLS configured"} [e2e-llm-inference-service] {"level":"info","ts":1785174383.0086446,"logger":"proxy server on port 8000","msg":"starting","addr":"[::]:8000"} [e2e-llm-inference-service] {"level":"info","ts":1785174383.66019,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174384.6679301,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174400.2820544,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174412.9217205,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785174412.9320621,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174412.9321115,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174442.9244368,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785174442.935343,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174442.9353995,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174472.9245083,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785174472.9358068,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174472.9358647,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174500.2873025,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174500.3068805,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174502.9251337,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785174502.935002,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174502.93505,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174532.9256663,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785174532.9365168,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174532.9365644,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174562.9260697,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785174562.936719,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174562.936772,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174592.926316,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785174592.9369767,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174592.9370303,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174622.926537,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785174622.9375093,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174622.937566,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174652.92692,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785174652.936796,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174652.9368446,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174682.927089,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785174682.9372826,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174682.9373307,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174712.9280946,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785174712.9382417,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174712.9382951,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174742.9284232,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785174742.940837,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174742.9408884,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174772.9292746,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785174772.939,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174772.939054,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174802.9302692,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785174802.9402835,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174802.9403334,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174832.93107,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785174832.9403749,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174832.94043,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174862.9319866,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785174862.942319,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174862.942371,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174892.9330738,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785174892.9433532,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174892.9434106,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174922.9334247,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785174922.9466825,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174922.9467432,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174952.9335651,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785174952.9426053,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174952.9426515,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174982.9346418,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785174982.944674,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174982.944743,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785175012.9348707,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785175012.948781,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785175012.9488285,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785175042.935804,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785175042.9455595,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785175042.9456093,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785175072.936001,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785175072.9452522,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785175072.9453115,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785175102.936951,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785175102.9468508,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785175102.9469042,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785175132.9375246,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785175132.9473352,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785175132.9473903,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785175162.9377763,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785175162.9478776,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785175162.9479296,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785175192.938158,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785175192.9495409,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785175192.9495928,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785175222.9385421,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785175222.9486043,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785175222.9486663,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785175252.9393165,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785175252.948575,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785175252.9486217,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.64":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582":{}}} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 (EngineCore pid=74) DEBUG 07-27 17:47:34 [compilation/decorators.py:213] Inferred dynamic dimensions for forward method of : ['num_tokens_no_spec', 'token_ids_gpu', 'combined_mask'] [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-27 17:47:34 [v1/worker/cpu_worker.py:236] auto thread-binding list (id, physical core): [(4, 0), (5, 1), (6, 2), (7, 3)] [e2e-llm-inference-service] [W727 17:47:34.281291396 utils.cpp:76] Warning: numa_migrate_pages failed. errno: 1 (function init_cpu_threads_env) [e2e-llm-inference-service] [W727 17:47:34.281317256 utils.cpp:103] Warning: NUMA binding: Using MEMBIND policy for memory allocation on the NUMA nodes (0). Memory allocations will be strictly bound to these NUMA nodes. (function init_cpu_threads_env) [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-27 17:47:34 [v1/worker/cpu_worker.py:109] OMP threads binding of Process 74: [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-27 17:47:34 [v1/worker/cpu_worker.py:109] OMP tid: 74, core 4 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-27 17:47:34 [v1/worker/cpu_worker.py:109] OMP tid: 91, core 5 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-27 17:47:34 [v1/worker/cpu_worker.py:109] OMP tid: 92, core 6 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-27 17:47:34 [v1/worker/cpu_worker.py:109] OMP tid: 93, core 7 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-27 17:47:34 [v1/worker/cpu_worker.py:109] [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-27 17:47:34 [distributed/parallel_state.py:1356] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.132.0.64:40357 backend=gloo [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-27 17:47:34 [distributed/parallel_state.py:1400] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.132.0.64:40357 backend=gloo [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-27 17:47:34 [distributed/parallel_state.py:1459] Detected 1 nodes in the distributed environment [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-27 17:47:34 [distributed/parallel_state.py:1716] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-27 17:47:34 [v1/sample/logits_processor/__init__.py:65] No logitsprocs plugins installed (group vllm.logits_processors). [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-27 17:47:34 [model_executor/offloader/base.py:107] Offloader set to NoopOffloader (no offloading). [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-27 17:47:34 [v1/worker/cpu_model_runner.py:71] Starting to load model /mnt/models... [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-27 17:47:34 [compilation/decorators.py:213] Inferred dynamic dimensions for forward method of : ['input_ids', 'positions', 'intermediate_tensors', 'inputs_embeds'] [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-27 17:47:34 [config/compilation.py:1194] enabled custom ops: Counter() [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-27 17:47:34 [config/compilation.py:1195] disabled custom ops: Counter({'vocab_parallel_embedding': 1, 'logits_processor': 1}) [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-27 17:47:34 [model_executor/model_loader/base_loader.py:63] Loading weights on cpu ... [e2e-llm-inference-service] (EngineCore pid=74) Loading pt checkpoint shards: 0% Completed | 0/1 [00:00 [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:47:43 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:47:53 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:48:03 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start. [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-27 17:48:12 [compilation/decorators.py:640] saved AOT compiled function to /home/.cache/vllm/torch_compile_cache/torch_aot_compile/86c9c3c579382eef68a98ac1d59b39811ba08abef3b4e90675a32c8dec3d7c90/rank_0_0/model [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-27 17:48:13 [compilation/monitor.py:76] Initial profiling/warmup run took 1.11 s [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-27 17:48:13 [v1/worker/cpu_model_runner.py:92] Warming up done. [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-27 17:48:13 [v1/engine/core.py:283] init engine (profile, create kv cache, warmup model) took 38.06 seconds [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-27 17:48:13 [tokenizers/registry.py:68] Loading CachedHfTokenizer for tokenizer_mode='hf' [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:48:13 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start. [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-27 17:48:14 [utils/gc_utils.py:40] GC Debug Config. enabled:False,top_objects:-1 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-27 17:48:14 [config/vllm.py:790] Asynchronous scheduling is disabled. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:48:14 [v1/engine/utils.py:1158] READY from local core engine process 0. [e2e-llm-inference-service] (EngineCore pid=74) WARNING 07-27 17:48:14 [config/vllm.py:859] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-27 17:48:14 [v1/engine/core.py:1158] EngineCore waiting for work. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:48:14 [v1/metrics/loggers.py:273] Engine 000: vllm cache_config_info with initialization after num_gpu_blocks is: 227 [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-27 17:48:14 [v1/engine/core.py:1158] EngineCore waiting for work. [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-27 17:48:14 [v1/engine/core.py:1158] EngineCore waiting for work. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/openai/api_server.py:590] Supported tasks: ['generate'] [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:48:14 [renderers/base.py:197] Warming up chat template processing... [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:48:14 [renderers/hf.py:122] Failed to load AutoTokenizer chat template for /mnt/models [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:48:14 [renderers/hf.py:122] Traceback (most recent call last): [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:48:14 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/vllm/renderers/hf.py", line 120, in resolve_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:48:14 [renderers/hf.py:122] return tokenizer.get_chat_template(chat_template, tools=tools) [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:48:14 [renderers/hf.py:122] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:48:14 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1825, in get_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:48:14 [renderers/hf.py:122] raise ValueError( [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:48:14 [renderers/hf.py:122] ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:48:14 [renderers/hf.py:141] There is no chat template fallback for /mnt/models [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [renderers/hf.py:314] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:48:14 [renderers/hf.py:122] Failed to load AutoTokenizer chat template for /mnt/models [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:48:14 [renderers/hf.py:122] Traceback (most recent call last): [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:48:14 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/vllm/renderers/hf.py", line 120, in resolve_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:48:14 [renderers/hf.py:122] return tokenizer.get_chat_template(chat_template, tools=tools) [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:48:14 [renderers/hf.py:122] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:48:14 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1825, in get_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:48:14 [renderers/hf.py:122] raise ValueError( [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:48:14 [renderers/hf.py:122] ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:48:14 [renderers/base.py:205] This model does not support chat template. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/openai/api_server.py:594] Starting vLLM server on https://0.0.0.0:8001 [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/launcher.py:37] Available routes are: [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/launcher.py:46] Route: /openapi.json, Methods: HEAD, GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/launcher.py:46] Route: /docs, Methods: HEAD, GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/launcher.py:46] Route: /redoc, Methods: HEAD, GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/launcher.py:46] Route: /tokenize, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/launcher.py:46] Route: /detokenize, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/launcher.py:46] Route: /load, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/launcher.py:46] Route: /version, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/launcher.py:46] Route: /health, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/launcher.py:46] Route: /metrics, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/launcher.py:46] Route: /v1/models, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/launcher.py:46] Route: /ping, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/launcher.py:46] Route: /ping, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/launcher.py:46] Route: /invocations, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/launcher.py:46] Route: /v1/chat/completions, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/launcher.py:46] Route: /v1/responses, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/launcher.py:46] Route: /v1/completions, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/launcher.py:46] Route: /v1/messages, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/launcher.py:46] Route: /inference/v1/generate, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/launcher.py:46] Route: /scale_elastic_ep, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/launcher.py:46] Route: /v1/chat/completions/render, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/launcher.py:46] Route: /v1/completions/render, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO: Started server process [1] [e2e-llm-inference-service] (APIServer pid=1) INFO: Waiting for application startup. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:14 [entrypoints/ssl.py:60] SSLCertRefresher monitors files: ['/var/run/kserve/tls/tls.key', '/var/run/kserve/tls/tls.crt'] [e2e-llm-inference-service] (APIServer pid=1) INFO: Application startup complete. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:48:20 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:20 [entrypoints/ssl.py:64] File change detected: deleted - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:20 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:20 [entrypoints/ssl.py:64] File change detected: modified - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:20 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:20 [entrypoints/ssl.py:64] File change detected: modified - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:20 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:20 [entrypoints/ssl.py:64] File change detected: deleted - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] (APIServer pid=1) INFO 07-27 17:48:20 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:48:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:48:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:48:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:48:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:49:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:49:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:49:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:49:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:49:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:49:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:50:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:50:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:50:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:50:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:50:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:50:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:51:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:51:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:51:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:51:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:51:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:51:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:52:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:52:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:52:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:52:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:52:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:52:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:53:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:53:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:53:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:53:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:53:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:53:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:54:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:54:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:54:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:54:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:54:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:54:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:55:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:55:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:55:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:55:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:55:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:55:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:56:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:56:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:56:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:56:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:56:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:56:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:57:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:57:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:57:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:57:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:57:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:57:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:58:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:58:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:58:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:58:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:58:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:58:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:59:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:59:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:59:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:59:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:59:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 17:59:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:00:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:00:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:00:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:00:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-27 18:00:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:228 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '3c42d2e7-a6e2-4e1d-bbe0-fbeb6a04fb5c', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Mon, 27 Jul 2026 18:00:53 GMT', 'Content-Length': '247'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb\" is waiting to start: ContainerCreating","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7 (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 {"level":"info","ts":1785174376.8209968,"logger":"setup","caller":"runner/runner.go:196","msg":"GIE build","commit-sha":"unknown","build-ref":""} [e2e-llm-inference-service] Flag --model-server-metrics-scheme has been deprecated, This flag is deprecated. Configure via EndpointPickerConfig data layer plugin parameters instead. [e2e-llm-inference-service] {"level":"info","ts":1785174376.821118,"logger":"setup","caller":"runner/runner.go:217","msg":"Flags processed","flags":{"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n- parameters:\n deciders:\n prefill: always-disagg-pd-decider\n type: disagg-profile-handler\nschedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef: active-request-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"https","pool-group":"inference.networking.k8s.io","pool-name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","pool-namespace":"e2e-test-llm-inference-service-48639af5","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8212469,"logger":"setup.trace","caller":"tracing/telemetry.go:123","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8218725,"caller":"loader/configloader.go:121","msg":"Loaded raw configuration","config":"{Plugins: [{Type: disagg-headers-handler} {Type: prefill-filter} {Type: decode-filter} {Type: queue-scorer} {Type: kv-cache-utilization-scorer} {Type: active-request-scorer} {Type: prefix-cache-scorer} {Type: max-score-picker} {Type: always-disagg-pd-decider} {Type: disagg-profile-handler, Parameters: {\"deciders\":{\"prefill\":\"always-disagg-pd-decider\"}}}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: active-request-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}]}"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8219006,"logger":"setup","caller":"runner/runner.go:622","msg":"Data layer: ENABLED"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8221803,"logger":"setup","caller":"runner/runner.go:281","msg":"Raw config after phase one","config":{"apiVersion":"llm-d.ai/v1alpha1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"disagg-headers-handler","parameters":null,"type":"disagg-headers-handler"},{"name":"prefill-filter","parameters":null,"type":"prefill-filter"},{"name":"decode-filter","parameters":null,"type":"decode-filter"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"kv-cache-utilization-scorer","parameters":null,"type":"kv-cache-utilization-scorer"},{"name":"active-request-scorer","parameters":null,"type":"active-request-scorer"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"},{"name":"always-disagg-pd-decider","parameters":null,"type":"always-disagg-pd-decider"},{"name":"disagg-profile-handler","parameters":{"deciders":{"prefill":"always-disagg-pd-decider"}},"type":"disagg-profile-handler"}],"schedulingProfiles":[{"name":"prefill","plugins":[{"pluginRef":"prefill-filter","weight":null},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"kv-cache-utilization-scorer","weight":2},{"pluginRef":"max-score-picker","weight":null}]},{"name":"decode","plugins":[{"pluginRef":"decode-filter","weight":null},{"pluginRef":"active-request-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]}]}} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8408985,"caller":"disagg/disagg_profile_handler.go:186","msg":"No deciders.encode configured, E disaggregation disabled"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8409505,"logger":"utilization-detector/utilization-detector","caller":"utilization/detector.go:83","msg":"Creating new UtilizationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms","headroom":0} [e2e-llm-inference-service] {"level":"info","ts":1785174376.841013,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"vllm","mapping":"Mapping{all specs enabled}"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.841047,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"sglang","mapping":"Mapping{disabled: [lora]}"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8410828,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"trtllm-serve","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8411434,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton-tensorrt-llm","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.841163,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton","mapping":"Mapping{disabled: [kv, lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.84123,"caller":"loader/configloader.go:154","msg":"Instantiated all plugins and applied system defaults. Effective raw configuration","config":"{Plugins: [{Name: disagg-headers-handler, Type: disagg-headers-handler} {Name: prefill-filter, Type: prefill-filter} {Name: decode-filter, Type: decode-filter} {Name: queue-scorer, Type: queue-scorer} {Name: kv-cache-utilization-scorer, Type: kv-cache-utilization-scorer} {Name: active-request-scorer, Type: active-request-scorer} {Name: prefix-cache-scorer, Type: prefix-cache-scorer} {Name: max-score-picker, Type: max-score-picker} {Name: always-disagg-pd-decider, Type: always-disagg-pd-decider} {Name: disagg-profile-handler, Type: disagg-profile-handler, Parameters: {\"deciders\":{\"prefill\":\"always-disagg-pd-decider\"}}} {Name: fcfs-ordering-policy, Type: fcfs-ordering-policy} {Name: global-strict-fairness-policy, Type: global-strict-fairness-policy} {Name: static-usage-limit-policy, Type: static-usage-limit-policy} {Name: openai-parser, Type: openai-parser} {Name: anthropic-parser, Type: anthropic-parser} {Name: vllmhttp-parser, Type: vllmhttp-parser} {Name: utilization-detector, Type: utilization-detector} {Name: metrics-data-source, Type: metrics-data-source} {Name: core-metrics-extractor, Type: core-metrics-extractor}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: active-request-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}], DataLayer: {Sources: [{PluginRef: metrics-data-source, Extractors: [{PluginRef: core-metrics-extractor}]}], Discovery: }, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}]}}"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8413136,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"inflight-load-producer/inflight-load-producer","dataKey":"InFlightLoadDataKey/inflight-load-producer","consumer":"active-request-scorer"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8413296,"caller":"approximateprefix/plugin.go:88","msg":"Prefix DataProducer initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":2048,"maxPrefixTokensToMatch":131072,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8413744,"caller":"approximateprefix/plugin.go:111","msg":"WARNING: configured blockSizeTokens is below the recommended minimum, overriding it.","blockSizeTokens":16,"minimum":64,"issue":"https://github.com/llm-d/llm-d-router/issues/1158"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8413877,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"approx-prefix-cache-producer/approx-prefix-cache-producer","dataKey":"PrefixCacheMatchInfoDataKey/approx-prefix-cache-producer","consumer":"disagg-profile-handler"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8413968,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"token-producer/token-producer","dataKey":"TokenizedPrompt/token-producer","consumer":"disagg-profile-handler"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8415277,"caller":"runner/runner.go:685","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8415387,"logger":"setup","caller":"runner/runner.go:308","msg":"EPP config after phase two","config":"{SchedulerConfig:{ProfileHandler: disagg-profile-handler/disagg-profile-handler, Profiles: map[decode:{Filters: [decode-filter/by-label], Scorers: [active-request-scorer/active-request-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker} prefill:{Filters: [prefill-filter/by-label], Scorers: [prefix-cache-scorer/prefix-cache-scorer: 3.000000, queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]} SaturationDetector:0xc0006fa4c0 DataConfig:{Sources:[{Plugin:0xc0006eabd0 Extractors:[0xc0006fa6c0]}]} FlowControlConfig: ParserRegistry:0xc0006fad40}"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8624604,"logger":"setup","caller":"runner/runner.go:352","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8625169,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8625329,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.862538,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8625424,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8625462,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8625503,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8625543,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/cmdline"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8625581,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/profile"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.862563,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/symbol"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8625667,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/trace"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8625708,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8625774,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/plugins/state"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8625815,"logger":"setup","caller":"runner/runner.go:373","msg":"parsed config","scheduler-config":"{ProfileHandler: disagg-profile-handler/disagg-profile-handler, Profiles: map[decode:{Filters: [decode-filter/by-label], Scorers: [active-request-scorer/active-request-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker} prefill:{Filters: [prefill-filter/by-label], Scorers: [prefix-cache-scorer/prefix-cache-scorer: 3.000000, queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8626187,"logger":"setup","caller":"datalayer/runtime.go:99","msg":"Configuring datalayer runtime","numSources":1} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8626273,"logger":"setup","caller":"datalayer/runtime.go:118","msg":"Processing source","source":"metrics-data-source","numExtractors":1} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8626423,"logger":"setup","caller":"datalayer/runtime.go:147","msg":"Source configured","source":"metrics-data-source","extractors":["core-metrics-extractor/core-metrics-extractor"]} [e2e-llm-inference-service] {"level":"info","ts":1785174376.862663,"logger":"setup","caller":"datalayer/runtime.go:206","msg":"Datalayer runtime configured","pollers":1,"notifiers":0,"endpointSources":1} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8626735,"logger":"setup","caller":"runner/runner.go:833","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8627539,"logger":"setup","caller":"runner/runner.go:721","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8627682,"logger":"setup","caller":"runner/runner.go:260","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.862799,"logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8630655,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8633852,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8644278,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.864748,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8649344,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.865365,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8657207,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8659205,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":1785174376.864053,"logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8694634,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.869632,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8696623,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.8702853,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.9663808,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.9663944,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1785174376.9664118,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785174376.966416,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785174377.0656986,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1785174377.065728,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785174377.0658858,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","namespace":"e2e-test-llm-inference-service-48639af5"},"namespace":"e2e-test-llm-inference-service-48639af5","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","reconcileID":"7e062e8a-ecf4-4a34-8952-5c49bfd9d69d"} [e2e-llm-inference-service] {"level":"info","ts":1785174377.066755,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":1785174377.066785,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785174387.8265796,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","namespace":"e2e-test-llm-inference-service-48639af5"},"namespace":"e2e-test-llm-inference-service-48639af5","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","reconcileID":"95649aca-dca3-4886-9631-3c3b5346d5f7"} [e2e-llm-inference-service] {"level":"info","ts":1785174500.3052053,"caller":"inflightload/producer.go:308","msg":"Injected dynamic attribute into endpoint","key":"InFlightLoadDataKey/inflight-load-producer","endpoint":"e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582-rank-0"} [e2e-llm-inference-service] {"level":"info","ts":1785174500.3052566,"caller":"controller/pod_reconciler.go:99","msg":"Pod already exists","controller":"pod","controllerGroup":"","controllerKind":"Pod","Pod":{"name":"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582","namespace":"e2e-test-llm-inference-service-48639af5"},"namespace":"e2e-test-llm-inference-service-48639af5","name":"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582","reconcileID":"9c802569-3db7-474a-b88c-5a104e601270"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: d9b387b8-0fca-41be-852b-cc28ba055bcc [e2e-llm-inference-service] resourceVersion: '58675' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-27T17:46:48Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.134.0.58 [e2e-llm-inference-service] nodeName: ip-10-0-132-194.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7 [e2e-llm-inference-service] uid: a5aafbfc-97c9-416a-89a1-97d6ca461fb0 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: a3a5b370-0143-4b38-9f18-c98afff27fe6 [e2e-llm-inference-service] resourceVersion: '59631' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-27T17:48:20Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T17:48:20Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.132.0.64 [e2e-llm-inference-service] nodeName: ip-10-0-143-86.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582 [e2e-llm-inference-service] uid: 232e8c6b-3861-4cf4-b14c-cd5777e84c58 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582 [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499f- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 232e8c6b-3861-4cf4-b14c-cd5777e84c58 [e2e-llm-inference-service] resourceVersion: '59628' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 86c954499f [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.64/23"],"mac_address":"0a:58:0a:84:00:40","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.64/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.132.0.64\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:84:00:40\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499f [e2e-llm-inference-service] uid: 5042cd6a-e412-4404-8944-554ad4cff65f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:15Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5042cd6a-e412-4404-8944-554ad4cff65f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: ip-10-0-143-86 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:20Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T17:48:20Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.132.0.64"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-5dlts [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --decoder-use-tls=true [e2e-llm-inference-service] - --prefiller-use-tls=true [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-5dlts [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000990000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+\ [e2e-llm-inference-service] \ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" != *\"\ [e2e-llm-inference-service] --kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"* ]] &&\ [e2e-llm-inference-service] \ [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\"\ [e2e-llm-inference-service] \n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"\ [e2e-llm-inference-service] facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-5dlts [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000990000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] nodeName: ip-10-0-143-86.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c31,c30 [e2e-llm-inference-service] fsGroup: 1000990000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-zh7c5 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-l4jvn [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T17:46:23Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T17:46:23Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T17:48:20Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T17:48:20Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] hostIP: 10.0.143.86 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.143.86 [e2e-llm-inference-service] podIP: 10.132.0.64 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.132.0.64 [e2e-llm-inference-service] startTime: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-27T17:46:22Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar@sha256:4d57f8fe65a63b44f345487af2b831e64ddf7fb01126d4f51cc8cf49b7412d00 [e2e-llm-inference-service] containerID: cri-o://49cfeb0e7e8481948c2362703626b137f72797d55b3dfe4bd1891154d2c791b7 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-5dlts [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000990000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000990000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-27T17:46:23Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: docker.io/vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: docker.io/vllm/vllm-openai-cpu@sha256:a8257c201fd2f696c146615987be61579aa2052367b73c27bd3fb1ab8e90bd4b [e2e-llm-inference-service] containerID: cri-o://ee76192acf3634911c7b5c4af7465ca46d15f421cac06afaabaf83df23421594 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-5dlts [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000990000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000990000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: c91913fd-4a7e-4848-9c1c-28950caab3bb [e2e-llm-inference-service] resourceVersion: '58065' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.57/23"],"mac_address":"0a:58:0a:86:00:39","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.57/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957 [e2e-llm-inference-service] uid: 00b84822-8172-4d62-b81b-9614e24feb0f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-132-194 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"00b84822-8172-4d62-b81b-9614e24feb0f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-22kmj [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+\ [e2e-llm-inference-service] \ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" != *\"\ [e2e-llm-inference-service] --kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"* ]] &&\ [e2e-llm-inference-service] \ [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\"\ [e2e-llm-inference-service] \n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"\ [e2e-llm-inference-service] facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-22kmj [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000990000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-132-194.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c31,c30 [e2e-llm-inference-service] fsGroup: 1000990000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-zh7c5 [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] hostIP: 10.0.132.194 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.132.194 [e2e-llm-inference-service] startTime: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: ContainerCreating [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-22kmj [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7 [e2e-llm-inference-service] generateName: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-789b8c94b- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: a5aafbfc-97c9-416a-89a1-97d6ca461fb0 [e2e-llm-inference-service] resourceVersion: '58674' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 789b8c94b [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.58/23"],"mac_address":"0a:58:0a:86:00:3a","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.58/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.58\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:3a\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-789b8c94b [e2e-llm-inference-service] uid: a8c6fb05-44c5-4b6a-8318-4cce41d1afa3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-132-194 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a8c6fb05-44c5-4b6a-8318-4cce41d1afa3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.58"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-z4hv7 [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type:\ [e2e-llm-inference-service] \ disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n- type:\ [e2e-llm-inference-service] \ queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n\ [e2e-llm-inference-service] - parameters:\n deciders:\n prefill: always-disagg-pd-decider\n type:\ [e2e-llm-inference-service] \ disagg-profile-handler\nschedulingProfiles:\n- name: prefill\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: prefill-filter\n - pluginRef: prefix-cache-scorer\n weight:\ [e2e-llm-inference-service] \ 3\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: kv-cache-utilization-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: decode-filter\n - pluginRef: active-request-scorer\n weight:\ [e2e-llm-inference-service] \ 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-z4hv7 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000990000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-132-194.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c31,c30 [e2e-llm-inference-service] fsGroup: 1000990000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-zh7c5 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-jg58z [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T17:46:17Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T17:46:48Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T17:46:48Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] hostIP: 10.0.132.194 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.132.194 [e2e-llm-inference-service] podIP: 10.134.0.58 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.58 [e2e-llm-inference-service] startTime: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-endpoint-picker@sha256:7fd1d61cb5505e026eebab0b9653758779c3dee241e37bea92a0bab961a48c50 [e2e-llm-inference-service] containerID: cri-o://3aaec604d8257b80c2eae719f99703b2b75d15ecbc774fb91b385b0f15bff37b [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-z4hv7 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000990000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000990000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 0f37903c-5a41-4cf7-b540-9a69fab7b3b3 [e2e-llm-inference-service] resourceVersion: '58058' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-jg58z [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-jg58z"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dd703244-c216-4ad9-9549-de5f7bae0f18"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-zh7c5"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-zh7c5 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-jg58z [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-zh7c5 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-jg58z [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: fe75b2e7-5bb3-4078-8e5c-111f37681775 [e2e-llm-inference-service] resourceVersion: '58015' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:15Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-l4jvn [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:15Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-l4jvn"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:15Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dd703244-c216-4ad9-9549-de5f7bae0f18"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-zh7c5"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-zh7c5 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-l4jvn [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-zh7c5 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-l4jvn [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: fd04635e-36dd-4786-a26e-9bdb92871688 [e2e-llm-inference-service] resourceVersion: '58080' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dd703244-c216-4ad9-9549-de5f7bae0f18"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.197.75 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.197.75 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 7cda6cb9-9416-4551-9f35-666d0c9b0c00 [e2e-llm-inference-service] resourceVersion: '58044' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dd703244-c216-4ad9-9549-de5f7bae0f18"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.172.14 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.172.14 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 43259275-d942-4cbe-bb6b-1261bf9788a0 [e2e-llm-inference-service] resourceVersion: '59634' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:15Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:15Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dd703244-c216-4ad9-9549-de5f7bae0f18"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-27T17:48:20Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --decoder-use-tls=true [e2e-llm-inference-service] - --prefiller-use-tls=true [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-27T17:48:20Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T17:48:20Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-27T17:48:20Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T17:46:15Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499f" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: ca1216f6-ec70-4963-96a1-3330da2e0020 [e2e-llm-inference-service] resourceVersion: '64452' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:15Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:15Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dd703244-c216-4ad9-9549-de5f7bae0f18"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-27T17:56:17Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:unavailableReplicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n\ [e2e-llm-inference-service] \ ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh\ [e2e-llm-inference-service] \ \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] unavailableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-27T17:56:17Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T17:56:17Z' [e2e-llm-inference-service] reason: ProgressDeadlineExceeded [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957" [e2e-llm-inference-service] has timed out progressing. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 861e127d-4e97-4f01-9355-713c2ff7ea4c [e2e-llm-inference-service] resourceVersion: '58678' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dd703244-c216-4ad9-9549-de5f7bae0f18"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n\ [e2e-llm-inference-service] - type: queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n\ [e2e-llm-inference-service] - parameters:\n deciders:\n prefill: always-disagg-pd-decider\n\ [e2e-llm-inference-service] \ type: disagg-profile-handler\nschedulingProfiles:\n- name: prefill\n\ [e2e-llm-inference-service] \ plugins:\n - pluginRef: prefill-filter\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ kv-cache-utilization-scorer\n weight: 2\n - pluginRef: max-score-picker\n\ [e2e-llm-inference-service] - name: decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef:\ [e2e-llm-inference-service] \ active-request-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-27T17:46:48Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T17:46:48Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-27T17:46:48Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-789b8c94b" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499f [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 5042cd6a-e412-4404-8944-554ad4cff65f [e2e-llm-inference-service] resourceVersion: '59633' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:15Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 86c954499f [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] uid: 43259275-d942-4cbe-bb6b-1261bf9788a0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:15Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"43259275-d942-4cbe-bb6b-1261bf9788a0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-27T17:48:20Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 86c954499f [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 86c954499f [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --decoder-use-tls=true [e2e-llm-inference-service] - --prefiller-use-tls=true [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 00b84822-8172-4d62-b81b-9614e24feb0f [e2e-llm-inference-service] resourceVersion: '58062' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] uid: ca1216f6-ec70-4963-96a1-3330da2e0020 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"ca1216f6-ec70-4963-96a1-3330da2e0020"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n\ [e2e-llm-inference-service] \ ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh\ [e2e-llm-inference-service] \ \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-789b8c94b [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: a8c6fb05-44c5-4b6a-8318-4cce41d1afa3 [e2e-llm-inference-service] resourceVersion: '58677' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 789b8c94b [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler [e2e-llm-inference-service] uid: 861e127d-4e97-4f01-9355-713c2ff7ea4c [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"861e127d-4e97-4f01-9355-713c2ff7ea4c"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 789b8c94b [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 789b8c94b [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n\ [e2e-llm-inference-service] - type: queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n\ [e2e-llm-inference-service] - parameters:\n deciders:\n prefill: always-disagg-pd-decider\n\ [e2e-llm-inference-service] \ type: disagg-profile-handler\nschedulingProfiles:\n- name: prefill\n\ [e2e-llm-inference-service] \ plugins:\n - pluginRef: prefill-filter\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ kv-cache-utilization-scorer\n weight: 2\n - pluginRef: max-score-picker\n\ [e2e-llm-inference-service] - name: decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef:\ [e2e-llm-inference-service] \ active-request-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: fb2e1760-1937-4e27-99e3-dd511a54ec60 [e2e-llm-inference-service] resourceVersion: '58074' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dd703244-c216-4ad9-9549-de5f7bae0f18"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 883928b4-bdae-43b2-9246-85b1be4b2dd3 [e2e-llm-inference-service] resourceVersion: '58022' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:15Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:15Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dd703244-c216-4ad9-9549-de5f7bae0f18"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: c3568c96-eb86-453e-8eb6-0649783d89b2 [e2e-llm-inference-service] resourceVersion: '58072' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dd703244-c216-4ad9-9549-de5f7bae0f18"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 2bb58081-244d-4d67-ac89-6cac6f3235e8 [e2e-llm-inference-service] resourceVersion: '58017' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:15Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:15Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dd703244-c216-4ad9-9549-de5f7bae0f18"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service-6glqk [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-epp-service- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 64847e2f-f71f-41cc-b249-aba76b8f1ec7 [e2e-llm-inference-service] resourceVersion: '58676' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-27T17:46:48Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] uid: fd04635e-36dd-4786-a26e-9bdb92871688 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"fd04635e-36dd-4786-a26e-9bdb92871688"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.58 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7 [e2e-llm-inference-service] uid: a5aafbfc-97c9-416a-89a1-97d6ca461fb0 [e2e-llm-inference-service] nodeName: ip-10-0-132-194.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-skpc6k [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 2033d657-7fca-4737-bbbf-a63bf55baad1 [e2e-llm-inference-service] resourceVersion: '59629' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-27T17:48:20Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] uid: 7cda6cb9-9416-4551-9f35-666d0c9b0c00 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-27T17:48:20Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7cda6cb9-9416-4551-9f35-666d0c9b0c00"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.132.0.64 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582 [e2e-llm-inference-service] uid: 232e8c6b-3861-4cf4-b14c-cd5777e84c58 [e2e-llm-inference-service] nodeName: ip-10-0-143-86.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: fb2e1760-1937-4e27-99e3-dd511a54ec60 [e2e-llm-inference-service] resourceVersion: '58074' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dd703244-c216-4ad9-9549-de5f7bae0f18"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-inference-service-48639af5:llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 883928b4-bdae-43b2-9246-85b1be4b2dd3 [e2e-llm-inference-service] resourceVersion: '58022' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:15Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:15Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dd703244-c216-4ad9-9549-de5f7bae0f18"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-inference-service-48639af5:llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: c3568c96-eb86-453e-8eb6-0649783d89b2 [e2e-llm-inference-service] resourceVersion: '58072' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dd703244-c216-4ad9-9549-de5f7bae0f18"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 2bb58081-244d-4d67-ac89-6cac6f3235e8 [e2e-llm-inference-service] resourceVersion: '58017' [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:15Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-27T17:46:15Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dd703244-c216-4ad9-9549-de5f7bae0f18"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dd703244-c216-4ad9-9549-de5f7bae0f18"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T17:46:27Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-27T17:46:27Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-27T17:46:28Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] resourceVersion: '58415' [e2e-llm-inference-service] uid: 455b7a5a-9e2c-47ea-8949-b35b558f1c5e [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T17:46:27Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T17:46:17Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dd703244-c216-4ad9-9549-de5f7bae0f18"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T17:46:27Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-27T17:46:27Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-27T17:46:28Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] resourceVersion: '58415' [e2e-llm-inference-service] uid: 455b7a5a-9e2c-47ea-8949-b35b558f1c5e [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T17:46:27Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T17:46:17Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dd703244-c216-4ad9-9549-de5f7bae0f18"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-27T17:46:27Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] resourceVersion: '58389' [e2e-llm-inference-service] uid: 77ff59d7-0607-43ed-a0db-f13e3b02228d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T17:46:27Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T17:46:27Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:18Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dd703244-c216-4ad9-9549-de5f7bae0f18"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T17:46:18Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-27T17:46:21Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] resourceVersion: '58283' [e2e-llm-inference-service] uid: 1f0fb545-c01c-4f30-96e1-fb45952439a9 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T17:46:19Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-27T17:46:21Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dd703244-c216-4ad9-9549-de5f7bae0f18"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] resourceVersion: '58114' [e2e-llm-inference-service] uid: 8c45d890-9afe-41a5-beef-927c971718d0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:28Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dd703244-c216-4ad9-9549-de5f7bae0f18"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T17:46:28Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] resourceVersion: '58402' [e2e-llm-inference-service] uid: fa2a7836-accf-4a61-9111-706bc8b60ca1 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dd703244-c216-4ad9-9549-de5f7bae0f18"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] resourceVersion: '58116' [e2e-llm-inference-service] uid: d2a90b3d-f9ce-4010-b807-9926ac234bba [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dd703244-c216-4ad9-9549-de5f7bae0f18"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] resourceVersion: '58114' [e2e-llm-inference-service] uid: 8c45d890-9afe-41a5-beef-927c971718d0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:28Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dd703244-c216-4ad9-9549-de5f7bae0f18"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T17:46:28Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] resourceVersion: '58402' [e2e-llm-inference-service] uid: fa2a7836-accf-4a61-9111-706bc8b60ca1 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dd703244-c216-4ad9-9549-de5f7bae0f18"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] resourceVersion: '58116' [e2e-llm-inference-service] uid: d2a90b3d-f9ce-4010-b807-9926ac234bba [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dd703244-c216-4ad9-9549-de5f7bae0f18"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] resourceVersion: '58114' [e2e-llm-inference-service] uid: 8c45d890-9afe-41a5-beef-927c971718d0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:28Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dd703244-c216-4ad9-9549-de5f7bae0f18"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T17:46:28Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] resourceVersion: '58402' [e2e-llm-inference-service] uid: fa2a7836-accf-4a61-9111-706bc8b60ca1 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dd703244-c216-4ad9-9549-de5f7bae0f18"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] resourceVersion: '58116' [e2e-llm-inference-service] uid: d2a90b3d-f9ce-4010-b807-9926ac234bba [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dd703244-c216-4ad9-9549-de5f7bae0f18"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-27T17:46:16Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: dd703244-c216-4ad9-9549-de5f7bae0f18 [e2e-llm-inference-service] resourceVersion: '58092' [e2e-llm-inference-service] uid: 0ad20b78-1839-4f37-9a81-dbaf4da04b84 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:00:53Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 86c954499f [e2e-llm-inference-service] timestamp: '2026-07-27T18:00:33Z' [e2e-llm-inference-service] window: 16.036s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 96105886n [e2e-llm-inference-service] memory: 2398984Ki [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 16136708n [e2e-llm-inference-service] memory: 22976Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] creationTimestamp: '2026-07-27T18:00:53Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 789b8c94b [e2e-llm-inference-service] timestamp: '2026-07-27T18:00:38Z' [e2e-llm-inference-service] window: 19.151s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 55114980n [e2e-llm-inference-service] memory: 36388Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-07-27T18:00:54.166552] end - ❌ 901.875s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-27T17:46:28Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:48:35Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:46:17Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:46:54Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:46:54Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:46:28Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-inference-service-48639af5 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_stop_feature[router-managed-workload-single-cpu-model-fb-opt-125m] __ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] name = 'stop-feature-test', namespace = 'e2e-test-llm-stop-feature-d9eef2a1' [e2e-llm-inference-service] version = 'v1alpha1' [e2e-llm-inference-service] [e2e-llm-inference-service] def get_llmisvc( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] name, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] version=constants.KSERVE_V1ALPHA1_VERSION, [e2e-llm-inference-service] ): [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return kserve_client.api_instance.get_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICE, [e2e-llm-inference-service] name, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1204: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-stop-feature-d9eef2a1' [e2e-llm-inference-service] plural = 'llminferenceservices', name = 'stop-feature-test' [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] [e2e-llm-inference-service] def get_namespaced_custom_object(self, group, version, namespace, plural, name, **kwargs): # noqa: E501 [e2e-llm-inference-service] """get_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Returns a namespace scoped custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.get_namespaced_custom_object(group, version, namespace, plural, name, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: the custom resource's group (required) [e2e-llm-inference-service] :param str version: the custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: the custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param str name: the custom object's name (required) [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: object [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] kwargs['_return_http_data_only'] = True [e2e-llm-inference-service] > return self.get_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, **kwargs) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:1632: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-stop-feature-d9eef2a1' [e2e-llm-inference-service] plural = 'llminferenceservices', name = 'stop-feature-test' [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] local_var_params = {'_return_http_data_only': True, 'all_params': ['group', 'version', 'namespace', 'plural', 'name', 'async_req', ...], 'auth_settings': ['BearerToken'], 'body_params': None, ...} [e2e-llm-inference-service] all_params = ['group', 'version', 'namespace', 'plural', 'name', 'async_req', ...] [e2e-llm-inference-service] key = '_return_http_data_only', val = True, collection_formats = {} [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'name': 'stop-feature-test', 'namespace': 'e2e-test-llm-stop-feature-d9eef2a1', 'plural': 'llminferenceservices', ...} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] [e2e-llm-inference-service] def get_namespaced_custom_object_with_http_info(self, group, version, namespace, plural, name, **kwargs): # noqa: E501 [e2e-llm-inference-service] """get_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Returns a namespace scoped custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.get_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: the custom resource's group (required) [e2e-llm-inference-service] :param str version: the custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: the custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param str name: the custom object's name (required) [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: tuple(object, status_code(int), headers(HTTPHeaderDict)) [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] local_var_params = locals() [e2e-llm-inference-service] [e2e-llm-inference-service] all_params = [ [e2e-llm-inference-service] 'group', [e2e-llm-inference-service] 'version', [e2e-llm-inference-service] 'namespace', [e2e-llm-inference-service] 'plural', [e2e-llm-inference-service] 'name' [e2e-llm-inference-service] ] [e2e-llm-inference-service] all_params.extend( [e2e-llm-inference-service] [ [e2e-llm-inference-service] 'async_req', [e2e-llm-inference-service] '_return_http_data_only', [e2e-llm-inference-service] '_preload_content', [e2e-llm-inference-service] '_request_timeout' [e2e-llm-inference-service] ] [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] for key, val in six.iteritems(local_var_params['kwargs']): [e2e-llm-inference-service] if key not in all_params: [e2e-llm-inference-service] raise ApiTypeError( [e2e-llm-inference-service] "Got an unexpected keyword argument '%s'" [e2e-llm-inference-service] " to method get_namespaced_custom_object" % key [e2e-llm-inference-service] ) [e2e-llm-inference-service] local_var_params[key] = val [e2e-llm-inference-service] del local_var_params['kwargs'] [e2e-llm-inference-service] # verify the required parameter 'group' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('group' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['group'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `group` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'version' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('version' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['version'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `version` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'namespace' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('namespace' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['namespace'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `namespace` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'plural' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('plural' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['plural'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `plural` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'name' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('name' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['name'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `name` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] collection_formats = {} [e2e-llm-inference-service] [e2e-llm-inference-service] path_params = {} [e2e-llm-inference-service] if 'group' in local_var_params: [e2e-llm-inference-service] path_params['group'] = local_var_params['group'] # noqa: E501 [e2e-llm-inference-service] if 'version' in local_var_params: [e2e-llm-inference-service] path_params['version'] = local_var_params['version'] # noqa: E501 [e2e-llm-inference-service] if 'namespace' in local_var_params: [e2e-llm-inference-service] path_params['namespace'] = local_var_params['namespace'] # noqa: E501 [e2e-llm-inference-service] if 'plural' in local_var_params: [e2e-llm-inference-service] path_params['plural'] = local_var_params['plural'] # noqa: E501 [e2e-llm-inference-service] if 'name' in local_var_params: [e2e-llm-inference-service] path_params['name'] = local_var_params['name'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] [e2e-llm-inference-service] header_params = {} [e2e-llm-inference-service] [e2e-llm-inference-service] form_params = [] [e2e-llm-inference-service] local_var_files = {} [e2e-llm-inference-service] [e2e-llm-inference-service] body_params = None [e2e-llm-inference-service] # HTTP header `Accept` [e2e-llm-inference-service] header_params['Accept'] = self.api_client.select_header_accept( [e2e-llm-inference-service] ['application/json']) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] # Authentication setting [e2e-llm-inference-service] auth_settings = ['BearerToken'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] > return self.api_client.call_api( [e2e-llm-inference-service] '/apis/{group}/{version}/namespaces/{namespace}/{plural}/{name}', 'GET', [e2e-llm-inference-service] path_params, [e2e-llm-inference-service] query_params, [e2e-llm-inference-service] header_params, [e2e-llm-inference-service] body=body_params, [e2e-llm-inference-service] post_params=form_params, [e2e-llm-inference-service] files=local_var_files, [e2e-llm-inference-service] response_type='object', # noqa: E501 [e2e-llm-inference-service] auth_settings=auth_settings, [e2e-llm-inference-service] async_req=local_var_params.get('async_req'), [e2e-llm-inference-service] _return_http_data_only=local_var_params.get('_return_http_data_only'), # noqa: E501 [e2e-llm-inference-service] _preload_content=local_var_params.get('_preload_content', True), [e2e-llm-inference-service] _request_timeout=local_var_params.get('_request_timeout'), [e2e-llm-inference-service] collection_formats=collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:1739: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/{group}/{version}/namespaces/{namespace}/{plural}/{name}' [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'name': 'stop-feature-test', 'namespace': 'e2e-test-llm-stop-feature-d9eef2a1', 'plural': 'llminferenceservices', ...} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], async_req = None, _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def call_api(self, resource_path, method, [e2e-llm-inference-service] path_params=None, query_params=None, header_params=None, [e2e-llm-inference-service] body=None, post_params=None, files=None, [e2e-llm-inference-service] response_type=None, auth_settings=None, async_req=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] """Makes the HTTP request (synchronous) and returns deserialized data. [e2e-llm-inference-service] [e2e-llm-inference-service] To make an async_req request, set the async_req parameter. [e2e-llm-inference-service] [e2e-llm-inference-service] :param resource_path: Path to method endpoint. [e2e-llm-inference-service] :param method: Method to call. [e2e-llm-inference-service] :param path_params: Path parameters in the url. [e2e-llm-inference-service] :param query_params: Query parameters in the url. [e2e-llm-inference-service] :param header_params: Header parameters to be [e2e-llm-inference-service] placed in the request header. [e2e-llm-inference-service] :param body: Request body. [e2e-llm-inference-service] :param post_params dict: Request post form parameters, [e2e-llm-inference-service] for `application/x-www-form-urlencoded`, `multipart/form-data`. [e2e-llm-inference-service] :param auth_settings list: Auth Settings names for the request. [e2e-llm-inference-service] :param response: Response data type. [e2e-llm-inference-service] :param files dict: key -> filename, value -> filepath, [e2e-llm-inference-service] for `multipart/form-data`. [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param collection_formats: dict of collection formats for path, query, [e2e-llm-inference-service] header, and post parameters. [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: [e2e-llm-inference-service] If async_req parameter is True, [e2e-llm-inference-service] the request will be called asynchronously. [e2e-llm-inference-service] The method will return the request thread. [e2e-llm-inference-service] If parameter async_req is False or missing, [e2e-llm-inference-service] then the method will return the response directly. [e2e-llm-inference-service] """ [e2e-llm-inference-service] if not async_req: [e2e-llm-inference-service] > return self.__call_api(resource_path, method, [e2e-llm-inference-service] path_params, query_params, header_params, [e2e-llm-inference-service] body, post_params, files, [e2e-llm-inference-service] response_type, auth_settings, [e2e-llm-inference-service] _return_http_data_only, collection_formats, [e2e-llm-inference-service] _preload_content, _request_timeout, _host) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:348: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-stop-feature-d9eef2a1/llminferenceservices/stop-feature-test' [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] path_params = [('group', 'serving.kserve.io'), ('version', 'v1alpha1'), ('namespace', 'e2e-test-llm-stop-feature-d9eef2a1'), ('plural', 'llminferenceservices'), ('name', 'stop-feature-test')] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def __call_api( [e2e-llm-inference-service] self, resource_path, method, path_params=None, [e2e-llm-inference-service] query_params=None, header_params=None, body=None, post_params=None, [e2e-llm-inference-service] files=None, response_type=None, auth_settings=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] [e2e-llm-inference-service] config = self.configuration [e2e-llm-inference-service] [e2e-llm-inference-service] # header parameters [e2e-llm-inference-service] header_params = header_params or {} [e2e-llm-inference-service] header_params.update(self.default_headers) [e2e-llm-inference-service] if self.cookie: [e2e-llm-inference-service] header_params['Cookie'] = self.cookie [e2e-llm-inference-service] if header_params: [e2e-llm-inference-service] header_params = self.sanitize_for_serialization(header_params) [e2e-llm-inference-service] header_params = dict(self.parameters_to_tuples(header_params, [e2e-llm-inference-service] collection_formats)) [e2e-llm-inference-service] [e2e-llm-inference-service] # path parameters [e2e-llm-inference-service] if path_params: [e2e-llm-inference-service] path_params = self.sanitize_for_serialization(path_params) [e2e-llm-inference-service] path_params = self.parameters_to_tuples(path_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] for k, v in path_params: [e2e-llm-inference-service] # specified safe chars, encode everything [e2e-llm-inference-service] resource_path = resource_path.replace( [e2e-llm-inference-service] '{%s}' % k, [e2e-llm-inference-service] quote(str(v), safe=config.safe_chars_for_path_param) [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # query parameters [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] query_params = self.sanitize_for_serialization(query_params) [e2e-llm-inference-service] query_params = self.parameters_to_tuples(query_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] # post parameters [e2e-llm-inference-service] if post_params or files: [e2e-llm-inference-service] post_params = post_params if post_params else [] [e2e-llm-inference-service] post_params = self.sanitize_for_serialization(post_params) [e2e-llm-inference-service] post_params = self.parameters_to_tuples(post_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] post_params.extend(self.files_parameters(files)) [e2e-llm-inference-service] [e2e-llm-inference-service] # auth setting [e2e-llm-inference-service] self.update_params_for_auth(header_params, query_params, auth_settings) [e2e-llm-inference-service] [e2e-llm-inference-service] # body [e2e-llm-inference-service] if body: [e2e-llm-inference-service] body = self.sanitize_for_serialization(body) [e2e-llm-inference-service] [e2e-llm-inference-service] # request url [e2e-llm-inference-service] if _host is None: [e2e-llm-inference-service] url = self.configuration.host + resource_path [e2e-llm-inference-service] else: [e2e-llm-inference-service] # use server/host defined in path or operation instead [e2e-llm-inference-service] url = _host + resource_path [e2e-llm-inference-service] [e2e-llm-inference-service] # perform request and return response [e2e-llm-inference-service] > response_data = self.request( [e2e-llm-inference-service] method, url, query_params=query_params, headers=header_params, [e2e-llm-inference-service] post_params=post_params, body=body, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:180: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] url = 'https://afb35e615998e4024955f70c61cd36d7-c1814657712f161b.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-stop-feature-d9eef2a1/llminferenceservices/stop-feature-test' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] post_params = [], body = None, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] post_params=None, body=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Makes the HTTP request using RESTClient.""" [e2e-llm-inference-service] if method == "GET": [e2e-llm-inference-service] > return self.rest_client.GET(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:373: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] url = 'https://afb35e615998e4024955f70c61cd36d7-c1814657712f161b.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-stop-feature-d9eef2a1/llminferenceservices/stop-feature-test' [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] query_params = [], _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def GET(self, url, headers=None, query_params=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] > return self.request("GET", url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] query_params=query_params) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:244: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] url = 'https://afb35e615998e4024955f70c61cd36d7-c1814657712f161b.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-stop-feature-d9eef2a1/llminferenceservices/stop-feature-test' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] body=None, post_params=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Perform requests. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: http request method [e2e-llm-inference-service] :param url: http request url [e2e-llm-inference-service] :param query_params: query parameters in the url [e2e-llm-inference-service] :param headers: http request headers [e2e-llm-inference-service] :param body: request json body, for `application/json` [e2e-llm-inference-service] :param post_params: request post parameters, [e2e-llm-inference-service] `application/x-www-form-urlencoded` [e2e-llm-inference-service] and `multipart/form-data` [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] """ [e2e-llm-inference-service] method = method.upper() [e2e-llm-inference-service] assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT', [e2e-llm-inference-service] 'PATCH', 'OPTIONS'] [e2e-llm-inference-service] [e2e-llm-inference-service] if post_params and body: [e2e-llm-inference-service] raise ApiValueError( [e2e-llm-inference-service] "body parameter cannot be used with post_params parameter." [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] post_params = post_params or {} [e2e-llm-inference-service] headers = headers or {} [e2e-llm-inference-service] [e2e-llm-inference-service] timeout = None [e2e-llm-inference-service] if _request_timeout: [e2e-llm-inference-service] if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)): # noqa: E501,F821 [e2e-llm-inference-service] timeout = urllib3.Timeout(total=_request_timeout) [e2e-llm-inference-service] elif (isinstance(_request_timeout, tuple) and [e2e-llm-inference-service] len(_request_timeout) == 2): [e2e-llm-inference-service] timeout = urllib3.Timeout( [e2e-llm-inference-service] connect=_request_timeout[0], read=_request_timeout[1]) [e2e-llm-inference-service] [e2e-llm-inference-service] if 'Content-Type' not in headers: [e2e-llm-inference-service] headers['Content-Type'] = 'application/json' [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE` [e2e-llm-inference-service] if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']: [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] url += '?' + urlencode(query_params) [e2e-llm-inference-service] if (re.search('json', headers['Content-Type'], re.IGNORECASE) or [e2e-llm-inference-service] headers['Content-Type'] == 'application/apply-patch+yaml'): [e2e-llm-inference-service] if headers['Content-Type'] == 'application/json-patch+json': [e2e-llm-inference-service] if not isinstance(body, list): [e2e-llm-inference-service] headers['Content-Type'] = \ [e2e-llm-inference-service] 'application/strategic-merge-patch+json' [e2e-llm-inference-service] request_body = None [e2e-llm-inference-service] if body is not None: [e2e-llm-inference-service] request_body = json.dumps(body) [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'application/x-www-form-urlencoded': # noqa: E501 [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=False, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'multipart/form-data': [e2e-llm-inference-service] # must del headers['Content-Type'], or the correct [e2e-llm-inference-service] # Content-Type which generated by urllib3 will be [e2e-llm-inference-service] # overwritten. [e2e-llm-inference-service] del headers['Content-Type'] [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=True, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] # Pass a `string` parameter directly in the body to support [e2e-llm-inference-service] # other content types than Json when `body` argument is [e2e-llm-inference-service] # provided in serialized form [e2e-llm-inference-service] elif isinstance(body, str) or isinstance(body, bytes): [e2e-llm-inference-service] request_body = body [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] else: [e2e-llm-inference-service] # Cannot generate the request from given parameters [e2e-llm-inference-service] msg = """Cannot prepare a request message for provided [e2e-llm-inference-service] arguments. Please check that your arguments match [e2e-llm-inference-service] declared content type.""" [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] # For `GET`, `HEAD` [e2e-llm-inference-service] else: [e2e-llm-inference-service] r = self.pool_manager.request(method, url, [e2e-llm-inference-service] fields=query_params, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] except urllib3.exceptions.SSLError as e: [e2e-llm-inference-service] msg = "{0}\n{1}".format(type(e).__name__, str(e)) [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] [e2e-llm-inference-service] if _preload_content: [e2e-llm-inference-service] r = RESTResponse(r) [e2e-llm-inference-service] [e2e-llm-inference-service] # In the python 3, the response.data is bytes. [e2e-llm-inference-service] # we need to decode it to string. [e2e-llm-inference-service] if six.PY3: [e2e-llm-inference-service] r.data = r.data.decode('utf8') [e2e-llm-inference-service] [e2e-llm-inference-service] # log response body [e2e-llm-inference-service] logger.debug("response body: %s", r.data) [e2e-llm-inference-service] [e2e-llm-inference-service] if not 200 <= r.status <= 299: [e2e-llm-inference-service] > raise ApiException(http_resp=r) [e2e-llm-inference-service] E kubernetes.client.exceptions.ApiException: (500) [e2e-llm-inference-service] E Reason: Internal Server Error [e2e-llm-inference-service] E HTTP response headers: HTTPHeaderDict({'Audit-Id': '64337e64-81a3-4a0c-a629-6414e567fddf', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'bf546a5c-3f60-4261-9b3f-c3b9e58fb8c8', 'X-Kubernetes-Pf-Prioritylevel-Uid': '39b8073e-7e81-4403-a6ca-a29718cbfcdd', 'Date': 'Mon, 27 Jul 2026 18:12:09 GMT', 'Content-Length': '330'}) [e2e-llm-inference-service] E HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"conversion webhook for serving.kserve.io/v1alpha2, Kind=LLMInferenceService failed: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/convert?timeout=30s\": no endpoints available for service \"llmisvc-webhook-server-service\"","code":500} [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:238: ApiException [e2e-llm-inference-service] [e2e-llm-inference-service] The above exception was the direct cause of the following exception: [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-fb-opt-125m'], prompt='KServe is a', service_name=... {'name': 'model-fb-opt-125m-stop-feature-8f213f2f'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="stop-feature-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_stop_feature(test_case: TestCase): [e2e-llm-inference-service] """Test that stopping an LLMInferenceService sets the Ready condition to False with reason Stopped.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Create the service [e2e-llm-inference-service] print(f"Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] [e2e-llm-inference-service] # Wait for the service to be ready [e2e-llm-inference-service] print(f"Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] print(f"✅ LLMInferenceService {service_name} is ready") [e2e-llm-inference-service] [e2e-llm-inference-service] # Stop the service by adding the stop annotation [e2e-llm-inference-service] print(f"Stopping LLMInferenceService {service_name}") [e2e-llm-inference-service] stop_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] [e2e-llm-inference-service] # Wait for the service to be marked as stopped [e2e-llm-inference-service] print(f"Waiting for LLMInferenceService {service_name} to be stopped") [e2e-llm-inference-service] wait_for_llm_isvc_stopped( [e2e-llm-inference-service] kserve_client, test_case.llm_service, timeout_seconds=120 [e2e-llm-inference-service] ) [e2e-llm-inference-service] print(f"✅ LLMInferenceService {service_name} is stopped") [e2e-llm-inference-service] [e2e-llm-inference-service] # Verify the workload resources are deleted [e2e-llm-inference-service] print(f"Verifying workload resources are deleted for {service_name}") [e2e-llm-inference-service] verify_workload_resources_deleted( [e2e-llm-inference-service] kserve_client, test_case.llm_service, timeout_seconds=120 [e2e-llm-inference-service] ) [e2e-llm-inference-service] print(f"✅ Workload resources deleted for {service_name}") [e2e-llm-inference-service] [e2e-llm-inference-service] # Restart the service by removing the stop annotation [e2e-llm-inference-service] print(f"Restarting LLMInferenceService {service_name}") [e2e-llm-inference-service] restart_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] [e2e-llm-inference-service] # Wait for the service to be ready again [e2e-llm-inference-service] print(f"Waiting for LLMInferenceService {service_name} to be ready again") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py:109: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...featur-e7f09208'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-stop-feature-8f213f2f'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-27T18:10:25.829626', start_time = 1785175825.8299575 [e2e-llm-inference-service] duration = 104.10943078994751, timestamp_end = '2026-07-27T18:12:09.939391' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....-stop-featur-e7f09208'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-stop-feature-8f213f2f'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f8e573d6ac0> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] > out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1339: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] name = 'stop-feature-test', namespace = 'e2e-test-llm-stop-feature-d9eef2a1' [e2e-llm-inference-service] version = 'v1alpha1' [e2e-llm-inference-service] [e2e-llm-inference-service] def get_llmisvc( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] name, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] version=constants.KSERVE_V1ALPHA1_VERSION, [e2e-llm-inference-service] ): [e2e-llm-inference-service] try: [e2e-llm-inference-service] return kserve_client.api_instance.get_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICE, [e2e-llm-inference-service] name, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except client.rest.ApiException as e: [e2e-llm-inference-service] > raise RuntimeError( [e2e-llm-inference-service] f"❌ Exception when calling CustomObjectsApi->" [e2e-llm-inference-service] f"get_namespaced_custom_object for LLMInferenceService: {e}" [e2e-llm-inference-service] ) from e [e2e-llm-inference-service] E RuntimeError: ❌ Exception when calling CustomObjectsApi->get_namespaced_custom_object for LLMInferenceService: (500) [e2e-llm-inference-service] E Reason: Internal Server Error [e2e-llm-inference-service] E HTTP response headers: HTTPHeaderDict({'Audit-Id': '64337e64-81a3-4a0c-a629-6414e567fddf', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'bf546a5c-3f60-4261-9b3f-c3b9e58fb8c8', 'X-Kubernetes-Pf-Prioritylevel-Uid': '39b8073e-7e81-4403-a6ca-a29718cbfcdd', 'Date': 'Mon, 27 Jul 2026 18:12:09 GMT', 'Content-Length': '330'}) [e2e-llm-inference-service] E HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"conversion webhook for serving.kserve.io/v1alpha2, Kind=LLMInferenceService failed: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/convert?timeout=30s\": no endpoints available for service \"llmisvc-webhook-server-service\"","code":500} [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1212: RuntimeError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-stop-feature-d9eef2a1 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-stop-feature-d9eef2a1 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-stop-feature-d9eef2a1 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-stop-feature-d9eef2a1 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-stop-feature-d9eef2a1 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-stop-feature-d9eef2a1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-stop-feature-tes-eeaa3ea3 in namespace e2e-test-llm-stop-feature-d9eef2a1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-stop-feature-tes-eeaa3ea3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-stop-feature-tes-eeaa3ea3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-single-cpu-stop-featur-e7f09208 in namespace e2e-test-llm-stop-feature-d9eef2a1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-single-cpu-stop-featur-e7f09208 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-single-cpu-stop-featur-e7f09208 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig model-fb-opt-125m-stop-feature-8f213f2f in namespace e2e-test-llm-stop-feature-d9eef2a1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig model-fb-opt-125m-stop-feature-8f213f2f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig model-fb-opt-125m-stop-feature-8f213f2f [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_stop_feature] [2026-07-27T18:06:57.756104] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-fb-opt-125m'], prompt='KServe is a', service_name='stop-feature-test', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-stop-feature-d9eef2a1', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'stop-feature-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-stop-feature-d9eef2a1', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-stop-feature-tes-eeaa3ea3'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-stop-featur-e7f09208'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-stop-feature-8f213f2f'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-27T18:06:57.769020] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'stop-feature-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-stop-feature-d9eef2a1', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-stop-feature-tes-eeaa3ea3'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-stop-featur-e7f09208'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-stop-feature-8f213f2f'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-27T18:06:58.028792] end - ✅ in 0.260s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-27T18:06:58.029063] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'stop-feature-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-stop-feature-d9eef2a1', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-stop-feature-tes-eeaa3ea3'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-stop-featur-e7f09208'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-stop-feature-8f213f2f'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-27T18:07:54Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:07:54Z', 'message': 'Inference Pool e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:07:54Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:07:54Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:07:54Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:07:54Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:07:54Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:07:54Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-27T18:08:35Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:08:35Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:08:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:07:54Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:08:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:08:35Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:08:35Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:08:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 11: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-27T17:14:03Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [wait_for_llm_isvc_ready] [2026-07-27T18:10:05.340925] end - ✅ in 187.312s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [stop_llmisvc] [2026-07-27T18:10:05.341090] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'stop-feature-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-stop-feature-d9eef2a1', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-stop-feature-tes-eeaa3ea3'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-stop-featur-e7f09208'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-stop-feature-8f213f2f'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [stop_llmisvc] [2026-07-27T18:10:06.029622] end - ✅ in 0.688s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_stopped] [2026-07-27T18:10:06.029790] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'stop-feature-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-stop-feature-d9eef2a1', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-stop-feature-tes-eeaa3ea3'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-stop-featur-e7f09208'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-stop-feature-8f213f2f'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={'timeout_seconds': 120} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Ready condition status is not False: True [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [wait_for_llm_isvc_stopped] [2026-07-27T18:10:24.329215] end - ✅ in 18.299s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [verify_workload_resources_deleted] [2026-07-27T18:10:24.329355] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'stop-feature-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-stop-feature-d9eef2a1', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-stop-feature-tes-eeaa3ea3'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-stop-featur-e7f09208'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-stop-feature-8f213f2f'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={'timeout_seconds': 120} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [verify_workload_resources_deleted] [2026-07-27T18:10:24.386624] end - ✅ in 0.057s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [restart_llmisvc] [2026-07-27T18:10:24.386833] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'stop-feature-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-stop-feature-d9eef2a1', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-stop-feature-tes-eeaa3ea3'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-stop-featur-e7f09208'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-stop-feature-8f213f2f'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [restart_llmisvc] [2026-07-27T18:10:25.829456] end - ✅ in 1.442s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-27T18:10:25.829626] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'stop-feature-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-stop-feature-d9eef2a1', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-stop-feature-tes-eeaa3ea3'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-stop-featur-e7f09208'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-stop-feature-8f213f2f'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'TokenizerReady', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'TokenizerReady', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-27T18:10:22Z', 'message': 'Service is stopped', 'reason': 'Stopped', 'severity': 'Info', 'status': 'False', 'type': 'GatewaysReady'}, {'lastTransitionTime': '2026-07-27T18:10:22Z', 'message': 'Service is stopped', 'reason': 'Stopped', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:10:22Z', 'message': 'Service is stopped', 'reason': 'Stopped', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:10:22Z', 'message': 'Service is stopped', 'reason': 'Stopped', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:10:22Z', 'message': 'Service is stopped', 'reason': 'Stopped', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:10:22Z', 'message': 'Service is stopped', 'reason': 'Stopped', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:07:54Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:10:22Z', 'message': 'Service is stopped', 'reason': 'Stopped', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:10:22Z', 'message': 'Service is stopped', 'reason': 'Stopped', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:10:22Z', 'message': 'Service is stopped', 'reason': 'Stopped', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:10:22Z', 'message': 'Service is stopped', 'reason': 'Stopped', 'severity': 'Info', 'status': 'False', 'type': 'TokenizerReady'}, {'lastTransitionTime': '2026-07-27T18:10:22Z', 'message': 'Service is stopped', 'reason': 'Stopped', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:10:22Z', 'message': 'Service is stopped', 'reason': 'Stopped', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'RouterReady', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-27T18:10:41Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:10:41Z', 'message': 'Inference Pool e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:10:41Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:07:54Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:10:41Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:10:41Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:10:41Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:10:41Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-27T18:11:14Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:11:14Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:11:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:07:54Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:11:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:11:14Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:11:14Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:11:14Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-27T18:12:09.939391] end - ❌ 104.109s: ❌ Exception when calling CustomObjectsApi->get_namespaced_custom_object for LLMInferenceService: (500) [e2e-llm-inference-service] Reason: Internal Server Error [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '64337e64-81a3-4a0c-a629-6414e567fddf', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'bf546a5c-3f60-4261-9b3f-c3b9e58fb8c8', 'X-Kubernetes-Pf-Prioritylevel-Uid': '39b8073e-7e81-4403-a6ca-a29718cbfcdd', 'Date': 'Mon, 27 Jul 2026 18:12:09 GMT', 'Content-Length': '330'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"conversion webhook for serving.kserve.io/v1alpha2, Kind=LLMInferenceService failed: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/convert?timeout=30s\": no endpoints available for service \"llmisvc-webhook-server-service\"","code":500} [e2e-llm-inference-service] [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_stop_feature] [2026-07-27T18:12:09.939591] end - ❌ 312.183s: ❌ Exception when calling CustomObjectsApi->get_namespaced_custom_object for LLMInferenceService: (500) [e2e-llm-inference-service] Reason: Internal Server Error [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '64337e64-81a3-4a0c-a629-6414e567fddf', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'bf546a5c-3f60-4261-9b3f-c3b9e58fb8c8', 'X-Kubernetes-Pf-Prioritylevel-Uid': '39b8073e-7e81-4403-a6ca-a29718cbfcdd', 'Date': 'Mon, 27 Jul 2026 18:12:09 GMT', 'Content-Length': '330'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"conversion webhook for serving.kserve.io/v1alpha2, Kind=LLMInferenceService failed: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/convert?timeout=30s\": no endpoints available for service \"llmisvc-webhook-server-service\"","code":500} [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-stop-feature-d9eef2a1 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _____________ test_llm_with_lora_adapters[single-lora-adapter-hf] ______________ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] llm_config = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-no-scheduler-lora-single-3cece2ef', 'namespace': 'e2e-test-llm-with-lora-adapters-3cbf5e9c'}, 'spec': {'router': {'route': {}}}} [e2e-llm-inference-service] namespace = 'e2e-test-llm-with-lora-adapters-3cbf5e9c' [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_or_update_llmisvc_config(kserve_client, llm_config, namespace=None): [e2e-llm-inference-service] """Create or update an LLMInferenceServiceConfig resource.""" [e2e-llm-inference-service] version = llm_config["apiVersion"].split("/")[1] [e2e-llm-inference-service] [e2e-llm-inference-service] if namespace is None: [e2e-llm-inference-service] namespace = llm_config.get("metadata", {}).get("namespace", "default") [e2e-llm-inference-service] [e2e-llm-inference-service] name = llm_config.get("metadata", {}).get("name") [e2e-llm-inference-service] if not name: [e2e-llm-inference-service] raise ValueError("LLMInferenceServiceConfig must have a name in metadata") [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Checking LLMInferenceServiceConfig {name} in namespace {namespace}") [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > existing_config = kserve_client.api_instance.get_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICECONFIG, [e2e-llm-inference-service] name, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/fixtures.py:1721: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-with-lora-adapters-3cbf5e9c' [e2e-llm-inference-service] plural = 'llminferenceserviceconfigs' [e2e-llm-inference-service] name = 'router-no-scheduler-lora-single-3cece2ef' [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] [e2e-llm-inference-service] def get_namespaced_custom_object(self, group, version, namespace, plural, name, **kwargs): # noqa: E501 [e2e-llm-inference-service] """get_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Returns a namespace scoped custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.get_namespaced_custom_object(group, version, namespace, plural, name, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: the custom resource's group (required) [e2e-llm-inference-service] :param str version: the custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: the custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param str name: the custom object's name (required) [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: object [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] kwargs['_return_http_data_only'] = True [e2e-llm-inference-service] > return self.get_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, **kwargs) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:1632: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-with-lora-adapters-3cbf5e9c' [e2e-llm-inference-service] plural = 'llminferenceserviceconfigs' [e2e-llm-inference-service] name = 'router-no-scheduler-lora-single-3cece2ef' [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] local_var_params = {'_return_http_data_only': True, 'all_params': ['group', 'version', 'namespace', 'plural', 'name', 'async_req', ...], 'auth_settings': ['BearerToken'], 'body_params': None, ...} [e2e-llm-inference-service] all_params = ['group', 'version', 'namespace', 'plural', 'name', 'async_req', ...] [e2e-llm-inference-service] key = '_return_http_data_only', val = True, collection_formats = {} [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'name': 'router-no-scheduler-lora-single-3cece2ef', 'namespace': 'e2e-test-llm-with-lora-adapters-3cbf5e9c', 'plural': 'llminferenceserviceconfigs', ...} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] [e2e-llm-inference-service] def get_namespaced_custom_object_with_http_info(self, group, version, namespace, plural, name, **kwargs): # noqa: E501 [e2e-llm-inference-service] """get_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Returns a namespace scoped custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.get_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: the custom resource's group (required) [e2e-llm-inference-service] :param str version: the custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: the custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param str name: the custom object's name (required) [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: tuple(object, status_code(int), headers(HTTPHeaderDict)) [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] local_var_params = locals() [e2e-llm-inference-service] [e2e-llm-inference-service] all_params = [ [e2e-llm-inference-service] 'group', [e2e-llm-inference-service] 'version', [e2e-llm-inference-service] 'namespace', [e2e-llm-inference-service] 'plural', [e2e-llm-inference-service] 'name' [e2e-llm-inference-service] ] [e2e-llm-inference-service] all_params.extend( [e2e-llm-inference-service] [ [e2e-llm-inference-service] 'async_req', [e2e-llm-inference-service] '_return_http_data_only', [e2e-llm-inference-service] '_preload_content', [e2e-llm-inference-service] '_request_timeout' [e2e-llm-inference-service] ] [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] for key, val in six.iteritems(local_var_params['kwargs']): [e2e-llm-inference-service] if key not in all_params: [e2e-llm-inference-service] raise ApiTypeError( [e2e-llm-inference-service] "Got an unexpected keyword argument '%s'" [e2e-llm-inference-service] " to method get_namespaced_custom_object" % key [e2e-llm-inference-service] ) [e2e-llm-inference-service] local_var_params[key] = val [e2e-llm-inference-service] del local_var_params['kwargs'] [e2e-llm-inference-service] # verify the required parameter 'group' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('group' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['group'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `group` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'version' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('version' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['version'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `version` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'namespace' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('namespace' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['namespace'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `namespace` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'plural' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('plural' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['plural'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `plural` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'name' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('name' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['name'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `name` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] collection_formats = {} [e2e-llm-inference-service] [e2e-llm-inference-service] path_params = {} [e2e-llm-inference-service] if 'group' in local_var_params: [e2e-llm-inference-service] path_params['group'] = local_var_params['group'] # noqa: E501 [e2e-llm-inference-service] if 'version' in local_var_params: [e2e-llm-inference-service] path_params['version'] = local_var_params['version'] # noqa: E501 [e2e-llm-inference-service] if 'namespace' in local_var_params: [e2e-llm-inference-service] path_params['namespace'] = local_var_params['namespace'] # noqa: E501 [e2e-llm-inference-service] if 'plural' in local_var_params: [e2e-llm-inference-service] path_params['plural'] = local_var_params['plural'] # noqa: E501 [e2e-llm-inference-service] if 'name' in local_var_params: [e2e-llm-inference-service] path_params['name'] = local_var_params['name'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] [e2e-llm-inference-service] header_params = {} [e2e-llm-inference-service] [e2e-llm-inference-service] form_params = [] [e2e-llm-inference-service] local_var_files = {} [e2e-llm-inference-service] [e2e-llm-inference-service] body_params = None [e2e-llm-inference-service] # HTTP header `Accept` [e2e-llm-inference-service] header_params['Accept'] = self.api_client.select_header_accept( [e2e-llm-inference-service] ['application/json']) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] # Authentication setting [e2e-llm-inference-service] auth_settings = ['BearerToken'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] > return self.api_client.call_api( [e2e-llm-inference-service] '/apis/{group}/{version}/namespaces/{namespace}/{plural}/{name}', 'GET', [e2e-llm-inference-service] path_params, [e2e-llm-inference-service] query_params, [e2e-llm-inference-service] header_params, [e2e-llm-inference-service] body=body_params, [e2e-llm-inference-service] post_params=form_params, [e2e-llm-inference-service] files=local_var_files, [e2e-llm-inference-service] response_type='object', # noqa: E501 [e2e-llm-inference-service] auth_settings=auth_settings, [e2e-llm-inference-service] async_req=local_var_params.get('async_req'), [e2e-llm-inference-service] _return_http_data_only=local_var_params.get('_return_http_data_only'), # noqa: E501 [e2e-llm-inference-service] _preload_content=local_var_params.get('_preload_content', True), [e2e-llm-inference-service] _request_timeout=local_var_params.get('_request_timeout'), [e2e-llm-inference-service] collection_formats=collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:1739: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/{group}/{version}/namespaces/{namespace}/{plural}/{name}' [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'name': 'router-no-scheduler-lora-single-3cece2ef', 'namespace': 'e2e-test-llm-with-lora-adapters-3cbf5e9c', 'plural': 'llminferenceserviceconfigs', ...} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], async_req = None, _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def call_api(self, resource_path, method, [e2e-llm-inference-service] path_params=None, query_params=None, header_params=None, [e2e-llm-inference-service] body=None, post_params=None, files=None, [e2e-llm-inference-service] response_type=None, auth_settings=None, async_req=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] """Makes the HTTP request (synchronous) and returns deserialized data. [e2e-llm-inference-service] [e2e-llm-inference-service] To make an async_req request, set the async_req parameter. [e2e-llm-inference-service] [e2e-llm-inference-service] :param resource_path: Path to method endpoint. [e2e-llm-inference-service] :param method: Method to call. [e2e-llm-inference-service] :param path_params: Path parameters in the url. [e2e-llm-inference-service] :param query_params: Query parameters in the url. [e2e-llm-inference-service] :param header_params: Header parameters to be [e2e-llm-inference-service] placed in the request header. [e2e-llm-inference-service] :param body: Request body. [e2e-llm-inference-service] :param post_params dict: Request post form parameters, [e2e-llm-inference-service] for `application/x-www-form-urlencoded`, `multipart/form-data`. [e2e-llm-inference-service] :param auth_settings list: Auth Settings names for the request. [e2e-llm-inference-service] :param response: Response data type. [e2e-llm-inference-service] :param files dict: key -> filename, value -> filepath, [e2e-llm-inference-service] for `multipart/form-data`. [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param collection_formats: dict of collection formats for path, query, [e2e-llm-inference-service] header, and post parameters. [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: [e2e-llm-inference-service] If async_req parameter is True, [e2e-llm-inference-service] the request will be called asynchronously. [e2e-llm-inference-service] The method will return the request thread. [e2e-llm-inference-service] If parameter async_req is False or missing, [e2e-llm-inference-service] then the method will return the response directly. [e2e-llm-inference-service] """ [e2e-llm-inference-service] if not async_req: [e2e-llm-inference-service] > return self.__call_api(resource_path, method, [e2e-llm-inference-service] path_params, query_params, header_params, [e2e-llm-inference-service] body, post_params, files, [e2e-llm-inference-service] response_type, auth_settings, [e2e-llm-inference-service] _return_http_data_only, collection_formats, [e2e-llm-inference-service] _preload_content, _request_timeout, _host) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:348: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-with-lora-adapters-3cbf5e9c/llminferenceserviceconfigs/router-no-scheduler-lora-single-3cece2ef' [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] path_params = [('group', 'serving.kserve.io'), ('version', 'v1alpha1'), ('namespace', 'e2e-test-llm-with-lora-adapters-3cbf5e9c'), ('plural', 'llminferenceserviceconfigs'), ('name', 'router-no-scheduler-lora-single-3cece2ef')] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def __call_api( [e2e-llm-inference-service] self, resource_path, method, path_params=None, [e2e-llm-inference-service] query_params=None, header_params=None, body=None, post_params=None, [e2e-llm-inference-service] files=None, response_type=None, auth_settings=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] [e2e-llm-inference-service] config = self.configuration [e2e-llm-inference-service] [e2e-llm-inference-service] # header parameters [e2e-llm-inference-service] header_params = header_params or {} [e2e-llm-inference-service] header_params.update(self.default_headers) [e2e-llm-inference-service] if self.cookie: [e2e-llm-inference-service] header_params['Cookie'] = self.cookie [e2e-llm-inference-service] if header_params: [e2e-llm-inference-service] header_params = self.sanitize_for_serialization(header_params) [e2e-llm-inference-service] header_params = dict(self.parameters_to_tuples(header_params, [e2e-llm-inference-service] collection_formats)) [e2e-llm-inference-service] [e2e-llm-inference-service] # path parameters [e2e-llm-inference-service] if path_params: [e2e-llm-inference-service] path_params = self.sanitize_for_serialization(path_params) [e2e-llm-inference-service] path_params = self.parameters_to_tuples(path_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] for k, v in path_params: [e2e-llm-inference-service] # specified safe chars, encode everything [e2e-llm-inference-service] resource_path = resource_path.replace( [e2e-llm-inference-service] '{%s}' % k, [e2e-llm-inference-service] quote(str(v), safe=config.safe_chars_for_path_param) [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # query parameters [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] query_params = self.sanitize_for_serialization(query_params) [e2e-llm-inference-service] query_params = self.parameters_to_tuples(query_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] # post parameters [e2e-llm-inference-service] if post_params or files: [e2e-llm-inference-service] post_params = post_params if post_params else [] [e2e-llm-inference-service] post_params = self.sanitize_for_serialization(post_params) [e2e-llm-inference-service] post_params = self.parameters_to_tuples(post_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] post_params.extend(self.files_parameters(files)) [e2e-llm-inference-service] [e2e-llm-inference-service] # auth setting [e2e-llm-inference-service] self.update_params_for_auth(header_params, query_params, auth_settings) [e2e-llm-inference-service] [e2e-llm-inference-service] # body [e2e-llm-inference-service] if body: [e2e-llm-inference-service] body = self.sanitize_for_serialization(body) [e2e-llm-inference-service] [e2e-llm-inference-service] # request url [e2e-llm-inference-service] if _host is None: [e2e-llm-inference-service] url = self.configuration.host + resource_path [e2e-llm-inference-service] else: [e2e-llm-inference-service] # use server/host defined in path or operation instead [e2e-llm-inference-service] url = _host + resource_path [e2e-llm-inference-service] [e2e-llm-inference-service] # perform request and return response [e2e-llm-inference-service] > response_data = self.request( [e2e-llm-inference-service] method, url, query_params=query_params, headers=header_params, [e2e-llm-inference-service] post_params=post_params, body=body, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:180: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] url = 'https://afb35e615998e4024955f70c61cd36d7-c1814657712f161b.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1a...amespaces/e2e-test-llm-with-lora-adapters-3cbf5e9c/llminferenceserviceconfigs/router-no-scheduler-lora-single-3cece2ef' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] post_params = [], body = None, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] post_params=None, body=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Makes the HTTP request using RESTClient.""" [e2e-llm-inference-service] if method == "GET": [e2e-llm-inference-service] > return self.rest_client.GET(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:373: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] url = 'https://afb35e615998e4024955f70c61cd36d7-c1814657712f161b.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1a...amespaces/e2e-test-llm-with-lora-adapters-3cbf5e9c/llminferenceserviceconfigs/router-no-scheduler-lora-single-3cece2ef' [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] query_params = [], _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def GET(self, url, headers=None, query_params=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] > return self.request("GET", url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] query_params=query_params) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:244: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] url = 'https://afb35e615998e4024955f70c61cd36d7-c1814657712f161b.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1a...amespaces/e2e-test-llm-with-lora-adapters-3cbf5e9c/llminferenceserviceconfigs/router-no-scheduler-lora-single-3cece2ef' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] body=None, post_params=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Perform requests. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: http request method [e2e-llm-inference-service] :param url: http request url [e2e-llm-inference-service] :param query_params: query parameters in the url [e2e-llm-inference-service] :param headers: http request headers [e2e-llm-inference-service] :param body: request json body, for `application/json` [e2e-llm-inference-service] :param post_params: request post parameters, [e2e-llm-inference-service] `application/x-www-form-urlencoded` [e2e-llm-inference-service] and `multipart/form-data` [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] """ [e2e-llm-inference-service] method = method.upper() [e2e-llm-inference-service] assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT', [e2e-llm-inference-service] 'PATCH', 'OPTIONS'] [e2e-llm-inference-service] [e2e-llm-inference-service] if post_params and body: [e2e-llm-inference-service] raise ApiValueError( [e2e-llm-inference-service] "body parameter cannot be used with post_params parameter." [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] post_params = post_params or {} [e2e-llm-inference-service] headers = headers or {} [e2e-llm-inference-service] [e2e-llm-inference-service] timeout = None [e2e-llm-inference-service] if _request_timeout: [e2e-llm-inference-service] if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)): # noqa: E501,F821 [e2e-llm-inference-service] timeout = urllib3.Timeout(total=_request_timeout) [e2e-llm-inference-service] elif (isinstance(_request_timeout, tuple) and [e2e-llm-inference-service] len(_request_timeout) == 2): [e2e-llm-inference-service] timeout = urllib3.Timeout( [e2e-llm-inference-service] connect=_request_timeout[0], read=_request_timeout[1]) [e2e-llm-inference-service] [e2e-llm-inference-service] if 'Content-Type' not in headers: [e2e-llm-inference-service] headers['Content-Type'] = 'application/json' [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE` [e2e-llm-inference-service] if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']: [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] url += '?' + urlencode(query_params) [e2e-llm-inference-service] if (re.search('json', headers['Content-Type'], re.IGNORECASE) or [e2e-llm-inference-service] headers['Content-Type'] == 'application/apply-patch+yaml'): [e2e-llm-inference-service] if headers['Content-Type'] == 'application/json-patch+json': [e2e-llm-inference-service] if not isinstance(body, list): [e2e-llm-inference-service] headers['Content-Type'] = \ [e2e-llm-inference-service] 'application/strategic-merge-patch+json' [e2e-llm-inference-service] request_body = None [e2e-llm-inference-service] if body is not None: [e2e-llm-inference-service] request_body = json.dumps(body) [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'application/x-www-form-urlencoded': # noqa: E501 [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=False, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'multipart/form-data': [e2e-llm-inference-service] # must del headers['Content-Type'], or the correct [e2e-llm-inference-service] # Content-Type which generated by urllib3 will be [e2e-llm-inference-service] # overwritten. [e2e-llm-inference-service] del headers['Content-Type'] [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=True, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] # Pass a `string` parameter directly in the body to support [e2e-llm-inference-service] # other content types than Json when `body` argument is [e2e-llm-inference-service] # provided in serialized form [e2e-llm-inference-service] elif isinstance(body, str) or isinstance(body, bytes): [e2e-llm-inference-service] request_body = body [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] else: [e2e-llm-inference-service] # Cannot generate the request from given parameters [e2e-llm-inference-service] msg = """Cannot prepare a request message for provided [e2e-llm-inference-service] arguments. Please check that your arguments match [e2e-llm-inference-service] declared content type.""" [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] # For `GET`, `HEAD` [e2e-llm-inference-service] else: [e2e-llm-inference-service] r = self.pool_manager.request(method, url, [e2e-llm-inference-service] fields=query_params, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] except urllib3.exceptions.SSLError as e: [e2e-llm-inference-service] msg = "{0}\n{1}".format(type(e).__name__, str(e)) [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] [e2e-llm-inference-service] if _preload_content: [e2e-llm-inference-service] r = RESTResponse(r) [e2e-llm-inference-service] [e2e-llm-inference-service] # In the python 3, the response.data is bytes. [e2e-llm-inference-service] # we need to decode it to string. [e2e-llm-inference-service] if six.PY3: [e2e-llm-inference-service] r.data = r.data.decode('utf8') [e2e-llm-inference-service] [e2e-llm-inference-service] # log response body [e2e-llm-inference-service] logger.debug("response body: %s", r.data) [e2e-llm-inference-service] [e2e-llm-inference-service] if not 200 <= r.status <= 299: [e2e-llm-inference-service] > raise ApiException(http_resp=r) [e2e-llm-inference-service] E kubernetes.client.exceptions.ApiException: (404) [e2e-llm-inference-service] E Reason: Not Found [e2e-llm-inference-service] E HTTP response headers: HTTPHeaderDict({'Audit-Id': 'c0434090-1fe6-4e3f-9264-42585d314ccd', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'bf546a5c-3f60-4261-9b3f-c3b9e58fb8c8', 'X-Kubernetes-Pf-Prioritylevel-Uid': '39b8073e-7e81-4403-a6ca-a29718cbfcdd', 'Date': 'Mon, 27 Jul 2026 18:12:42 GMT', 'Content-Length': '338'}) [e2e-llm-inference-service] E HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"llminferenceserviceconfigs.serving.kserve.io \"router-no-scheduler-lora-single-3cece2ef\" not found","reason":"NotFound","details":{"name":"router-no-scheduler-lora-single-3cece2ef","group":"serving.kserve.io","kind":"llminferenceserviceconfigs"},"code":404} [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:238: ApiException [e2e-llm-inference-service] [e2e-llm-inference-service] During handling of the above exception, another exception occurred: [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = LoRATestCase(base_refs=['router-no-scheduler', 'workload-single-cpu', 'model-fb-opt-125m-with-lora-hf'], prompt='What ...ice_name='lora-single-adapter-test', endpoint='/v1/completions', max_tokens=100, wait_timeout=900, response_timeout=60) [e2e-llm-inference-service] test_namespace = 'e2e-test-llm-with-lora-adapters-3cbf5e9c' [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] LoRATestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is Kubernetes?", [e2e-llm-inference-service] expected_adapter_names=["lora-adapter-1"], [e2e-llm-inference-service] service_name="lora-single-adapter-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] id="single-lora-adapter-hf", [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] LoRATestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-multiple-lora", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Explain machine learning in simple terms.", [e2e-llm-inference-service] expected_adapter_names=["lora-adapter-1", "lora-adapter-2"], [e2e-llm-inference-service] service_name="lora-multiple-adapters-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] id="multiple-lora-adapters", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_with_lora_adapters(test_case: LoRATestCase, test_namespace): [e2e-llm-inference-service] """Test LLMInferenceService with LoRA adapters.""" [e2e-llm-inference-service] > run_lora_test(test_case, namespace=test_namespace) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py:252: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = LoRATestCase(base_refs=['router-no-scheduler', 'workload-single-cpu', 'model-fb-opt-125m-with-lora-hf'], prompt='What ...ice_name='lora-single-adapter-test', endpoint='/v1/completions', max_tokens=100, wait_timeout=900, response_timeout=60) [e2e-llm-inference-service] namespace = 'e2e-test-llm-with-lora-adapters-3cbf5e9c' [e2e-llm-inference-service] [e2e-llm-inference-service] def run_lora_test(test_case: LoRATestCase, namespace: str): [e2e-llm-inference-service] """Execute a LoRA adapter test case.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] test_id = generate_test_id(test_case) [e2e-llm-inference-service] service_name = test_case.service_name or f"lora-test-{test_id}" [e2e-llm-inference-service] model_name = _get_model_name_from_configs(test_case.base_refs) [e2e-llm-inference-service] created_configs = [] [e2e-llm-inference-service] llm_service = None [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Create unique LLMInferenceServiceConfig resources for each base ref [e2e-llm-inference-service] unique_base_refs = [] [e2e-llm-inference-service] for base_ref in test_case.base_refs: [e2e-llm-inference-service] if base_ref not in LLMINFERENCESERVICE_CONFIGS: [e2e-llm-inference-service] raise ValueError(f"Unknown base reference: {base_ref}") [e2e-llm-inference-service] [e2e-llm-inference-service] # Generate unique config name to avoid conflicts in parallel test runs [e2e-llm-inference-service] unique_config_name = generate_k8s_safe_suffix(base_ref, [service_name]) [e2e-llm-inference-service] unique_base_refs.append(unique_config_name) [e2e-llm-inference-service] [e2e-llm-inference-service] config = LLMINFERENCESERVICE_CONFIGS[base_ref] [e2e-llm-inference-service] config_body = { [e2e-llm-inference-service] "apiVersion": "serving.kserve.io/v1alpha1", [e2e-llm-inference-service] "kind": "LLMInferenceServiceConfig", [e2e-llm-inference-service] "metadata": { [e2e-llm-inference-service] "name": unique_config_name, [e2e-llm-inference-service] "namespace": namespace, [e2e-llm-inference-service] }, [e2e-llm-inference-service] "spec": config, [e2e-llm-inference-service] } [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] "Creating LLMInferenceServiceConfig: %s", [e2e-llm-inference-service] unique_config_name, [e2e-llm-inference-service] ) [e2e-llm-inference-service] > _create_or_update_llmisvc_config(kserve_client, config_body, namespace) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py:122: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] llm_config = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-no-scheduler-lora-single-3cece2ef', 'namespace': 'e2e-test-llm-with-lora-adapters-3cbf5e9c'}, 'spec': {'router': {'route': {}}}} [e2e-llm-inference-service] namespace = 'e2e-test-llm-with-lora-adapters-3cbf5e9c' [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_or_update_llmisvc_config(kserve_client, llm_config, namespace=None): [e2e-llm-inference-service] """Create or update an LLMInferenceServiceConfig resource.""" [e2e-llm-inference-service] version = llm_config["apiVersion"].split("/")[1] [e2e-llm-inference-service] [e2e-llm-inference-service] if namespace is None: [e2e-llm-inference-service] namespace = llm_config.get("metadata", {}).get("namespace", "default") [e2e-llm-inference-service] [e2e-llm-inference-service] name = llm_config.get("metadata", {}).get("name") [e2e-llm-inference-service] if not name: [e2e-llm-inference-service] raise ValueError("LLMInferenceServiceConfig must have a name in metadata") [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Checking LLMInferenceServiceConfig {name} in namespace {namespace}") [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] existing_config = kserve_client.api_instance.get_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICECONFIG, [e2e-llm-inference-service] name, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llm_config["metadata"] = existing_config["metadata"] [e2e-llm-inference-service] [e2e-llm-inference-service] outputs = kserve_client.api_instance.replace_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICECONFIG, [e2e-llm-inference-service] name, [e2e-llm-inference-service] llm_config, [e2e-llm-inference-service] ) [e2e-llm-inference-service] logger.info(f"✓ Successfully updated LLMInferenceServiceConfig {name}") [e2e-llm-inference-service] return outputs [e2e-llm-inference-service] [e2e-llm-inference-service] except client.rest.ApiException as e: [e2e-llm-inference-service] if e.status == 404: # Not found - create it [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"Resource not found, creating LLMInferenceServiceConfig {name}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] > outputs = kserve_client.api_instance.create_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICECONFIG, [e2e-llm-inference-service] llm_config, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/fixtures.py:1747: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-with-lora-adapters-3cbf5e9c' [e2e-llm-inference-service] plural = 'llminferenceserviceconfigs' [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-no-scheduler-lora-single-3cece2ef', 'namespace': 'e2e-test-llm-with-lora-adapters-3cbf5e9c'}, 'spec': {'router': {'route': {}}}} [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] [e2e-llm-inference-service] def create_namespaced_custom_object(self, group, version, namespace, plural, body, **kwargs): # noqa: E501 [e2e-llm-inference-service] """create_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Creates a namespace scoped Custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.create_namespaced_custom_object(group, version, namespace, plural, body, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: The custom resource's group name (required) [e2e-llm-inference-service] :param str version: The custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: The custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param object body: The JSON schema of the Resource to create. (required) [e2e-llm-inference-service] :param str pretty: If 'true', then the output is pretty printed. [e2e-llm-inference-service] :param str dry_run: When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed [e2e-llm-inference-service] :param str field_manager: fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. [e2e-llm-inference-service] :param str field_validation: fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default in v1.23+ - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered. (optional) [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: object [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] kwargs['_return_http_data_only'] = True [e2e-llm-inference-service] > return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:231: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-with-lora-adapters-3cbf5e9c' [e2e-llm-inference-service] plural = 'llminferenceserviceconfigs' [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-no-scheduler-lora-single-3cece2ef', 'namespace': 'e2e-test-llm-with-lora-adapters-3cbf5e9c'}, 'spec': {'router': {'route': {}}}} [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] local_var_params = {'_return_http_data_only': True, 'all_params': ['group', 'version', 'namespace', 'plural', 'body', 'pretty', ...], 'au...ra-single-3cece2ef', 'namespace': 'e2e-test-llm-with-lora-adapters-3cbf5e9c'}, 'spec': {'router': {'route': {}}}}, ...} [e2e-llm-inference-service] all_params = ['group', 'version', 'namespace', 'plural', 'body', 'pretty', ...] [e2e-llm-inference-service] key = '_return_http_data_only', val = True, collection_formats = {} [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'namespace': 'e2e-test-llm-with-lora-adapters-3cbf5e9c', 'plural': 'llminferenceserviceconfigs', 'version': 'v1alpha1'} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] [e2e-llm-inference-service] def create_namespaced_custom_object_with_http_info(self, group, version, namespace, plural, body, **kwargs): # noqa: E501 [e2e-llm-inference-service] """create_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Creates a namespace scoped Custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: The custom resource's group name (required) [e2e-llm-inference-service] :param str version: The custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: The custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param object body: The JSON schema of the Resource to create. (required) [e2e-llm-inference-service] :param str pretty: If 'true', then the output is pretty printed. [e2e-llm-inference-service] :param str dry_run: When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed [e2e-llm-inference-service] :param str field_manager: fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. [e2e-llm-inference-service] :param str field_validation: fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default in v1.23+ - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered. (optional) [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: tuple(object, status_code(int), headers(HTTPHeaderDict)) [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] local_var_params = locals() [e2e-llm-inference-service] [e2e-llm-inference-service] all_params = [ [e2e-llm-inference-service] 'group', [e2e-llm-inference-service] 'version', [e2e-llm-inference-service] 'namespace', [e2e-llm-inference-service] 'plural', [e2e-llm-inference-service] 'body', [e2e-llm-inference-service] 'pretty', [e2e-llm-inference-service] 'dry_run', [e2e-llm-inference-service] 'field_manager', [e2e-llm-inference-service] 'field_validation' [e2e-llm-inference-service] ] [e2e-llm-inference-service] all_params.extend( [e2e-llm-inference-service] [ [e2e-llm-inference-service] 'async_req', [e2e-llm-inference-service] '_return_http_data_only', [e2e-llm-inference-service] '_preload_content', [e2e-llm-inference-service] '_request_timeout' [e2e-llm-inference-service] ] [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] for key, val in six.iteritems(local_var_params['kwargs']): [e2e-llm-inference-service] if key not in all_params: [e2e-llm-inference-service] raise ApiTypeError( [e2e-llm-inference-service] "Got an unexpected keyword argument '%s'" [e2e-llm-inference-service] " to method create_namespaced_custom_object" % key [e2e-llm-inference-service] ) [e2e-llm-inference-service] local_var_params[key] = val [e2e-llm-inference-service] del local_var_params['kwargs'] [e2e-llm-inference-service] # verify the required parameter 'group' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('group' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['group'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `group` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'version' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('version' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['version'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `version` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'namespace' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('namespace' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['namespace'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `namespace` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'plural' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('plural' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['plural'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `plural` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'body' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('body' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['body'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `body` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] collection_formats = {} [e2e-llm-inference-service] [e2e-llm-inference-service] path_params = {} [e2e-llm-inference-service] if 'group' in local_var_params: [e2e-llm-inference-service] path_params['group'] = local_var_params['group'] # noqa: E501 [e2e-llm-inference-service] if 'version' in local_var_params: [e2e-llm-inference-service] path_params['version'] = local_var_params['version'] # noqa: E501 [e2e-llm-inference-service] if 'namespace' in local_var_params: [e2e-llm-inference-service] path_params['namespace'] = local_var_params['namespace'] # noqa: E501 [e2e-llm-inference-service] if 'plural' in local_var_params: [e2e-llm-inference-service] path_params['plural'] = local_var_params['plural'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] if 'pretty' in local_var_params and local_var_params['pretty'] is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('pretty', local_var_params['pretty'])) # noqa: E501 [e2e-llm-inference-service] if 'dry_run' in local_var_params and local_var_params['dry_run'] is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('dryRun', local_var_params['dry_run'])) # noqa: E501 [e2e-llm-inference-service] if 'field_manager' in local_var_params and local_var_params['field_manager'] is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('fieldManager', local_var_params['field_manager'])) # noqa: E501 [e2e-llm-inference-service] if 'field_validation' in local_var_params and local_var_params['field_validation'] is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('fieldValidation', local_var_params['field_validation'])) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] header_params = {} [e2e-llm-inference-service] [e2e-llm-inference-service] form_params = [] [e2e-llm-inference-service] local_var_files = {} [e2e-llm-inference-service] [e2e-llm-inference-service] body_params = None [e2e-llm-inference-service] if 'body' in local_var_params: [e2e-llm-inference-service] body_params = local_var_params['body'] [e2e-llm-inference-service] # HTTP header `Accept` [e2e-llm-inference-service] header_params['Accept'] = self.api_client.select_header_accept( [e2e-llm-inference-service] ['application/json']) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] # Authentication setting [e2e-llm-inference-service] auth_settings = ['BearerToken'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] > return self.api_client.call_api( [e2e-llm-inference-service] '/apis/{group}/{version}/namespaces/{namespace}/{plural}', 'POST', [e2e-llm-inference-service] path_params, [e2e-llm-inference-service] query_params, [e2e-llm-inference-service] header_params, [e2e-llm-inference-service] body=body_params, [e2e-llm-inference-service] post_params=form_params, [e2e-llm-inference-service] files=local_var_files, [e2e-llm-inference-service] response_type='object', # noqa: E501 [e2e-llm-inference-service] auth_settings=auth_settings, [e2e-llm-inference-service] async_req=local_var_params.get('async_req'), [e2e-llm-inference-service] _return_http_data_only=local_var_params.get('_return_http_data_only'), # noqa: E501 [e2e-llm-inference-service] _preload_content=local_var_params.get('_preload_content', True), [e2e-llm-inference-service] _request_timeout=local_var_params.get('_request_timeout'), [e2e-llm-inference-service] collection_formats=collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:354: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/{group}/{version}/namespaces/{namespace}/{plural}' [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'namespace': 'e2e-test-llm-with-lora-adapters-3cbf5e9c', 'plural': 'llminferenceserviceconfigs', 'version': 'v1alpha1'} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-no-scheduler-lora-single-3cece2ef', 'namespace': 'e2e-test-llm-with-lora-adapters-3cbf5e9c'}, 'spec': {'router': {'route': {}}}} [e2e-llm-inference-service] post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], async_req = None, _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def call_api(self, resource_path, method, [e2e-llm-inference-service] path_params=None, query_params=None, header_params=None, [e2e-llm-inference-service] body=None, post_params=None, files=None, [e2e-llm-inference-service] response_type=None, auth_settings=None, async_req=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] """Makes the HTTP request (synchronous) and returns deserialized data. [e2e-llm-inference-service] [e2e-llm-inference-service] To make an async_req request, set the async_req parameter. [e2e-llm-inference-service] [e2e-llm-inference-service] :param resource_path: Path to method endpoint. [e2e-llm-inference-service] :param method: Method to call. [e2e-llm-inference-service] :param path_params: Path parameters in the url. [e2e-llm-inference-service] :param query_params: Query parameters in the url. [e2e-llm-inference-service] :param header_params: Header parameters to be [e2e-llm-inference-service] placed in the request header. [e2e-llm-inference-service] :param body: Request body. [e2e-llm-inference-service] :param post_params dict: Request post form parameters, [e2e-llm-inference-service] for `application/x-www-form-urlencoded`, `multipart/form-data`. [e2e-llm-inference-service] :param auth_settings list: Auth Settings names for the request. [e2e-llm-inference-service] :param response: Response data type. [e2e-llm-inference-service] :param files dict: key -> filename, value -> filepath, [e2e-llm-inference-service] for `multipart/form-data`. [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param collection_formats: dict of collection formats for path, query, [e2e-llm-inference-service] header, and post parameters. [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: [e2e-llm-inference-service] If async_req parameter is True, [e2e-llm-inference-service] the request will be called asynchronously. [e2e-llm-inference-service] The method will return the request thread. [e2e-llm-inference-service] If parameter async_req is False or missing, [e2e-llm-inference-service] then the method will return the response directly. [e2e-llm-inference-service] """ [e2e-llm-inference-service] if not async_req: [e2e-llm-inference-service] > return self.__call_api(resource_path, method, [e2e-llm-inference-service] path_params, query_params, header_params, [e2e-llm-inference-service] body, post_params, files, [e2e-llm-inference-service] response_type, auth_settings, [e2e-llm-inference-service] _return_http_data_only, collection_formats, [e2e-llm-inference-service] _preload_content, _request_timeout, _host) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:348: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-with-lora-adapters-3cbf5e9c/llminferenceserviceconfigs' [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] path_params = [('group', 'serving.kserve.io'), ('version', 'v1alpha1'), ('namespace', 'e2e-test-llm-with-lora-adapters-3cbf5e9c'), ('plural', 'llminferenceserviceconfigs')] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-no-scheduler-lora-single-3cece2ef', 'namespace': 'e2e-test-llm-with-lora-adapters-3cbf5e9c'}, 'spec': {'router': {'route': {}}}} [e2e-llm-inference-service] post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def __call_api( [e2e-llm-inference-service] self, resource_path, method, path_params=None, [e2e-llm-inference-service] query_params=None, header_params=None, body=None, post_params=None, [e2e-llm-inference-service] files=None, response_type=None, auth_settings=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] [e2e-llm-inference-service] config = self.configuration [e2e-llm-inference-service] [e2e-llm-inference-service] # header parameters [e2e-llm-inference-service] header_params = header_params or {} [e2e-llm-inference-service] header_params.update(self.default_headers) [e2e-llm-inference-service] if self.cookie: [e2e-llm-inference-service] header_params['Cookie'] = self.cookie [e2e-llm-inference-service] if header_params: [e2e-llm-inference-service] header_params = self.sanitize_for_serialization(header_params) [e2e-llm-inference-service] header_params = dict(self.parameters_to_tuples(header_params, [e2e-llm-inference-service] collection_formats)) [e2e-llm-inference-service] [e2e-llm-inference-service] # path parameters [e2e-llm-inference-service] if path_params: [e2e-llm-inference-service] path_params = self.sanitize_for_serialization(path_params) [e2e-llm-inference-service] path_params = self.parameters_to_tuples(path_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] for k, v in path_params: [e2e-llm-inference-service] # specified safe chars, encode everything [e2e-llm-inference-service] resource_path = resource_path.replace( [e2e-llm-inference-service] '{%s}' % k, [e2e-llm-inference-service] quote(str(v), safe=config.safe_chars_for_path_param) [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # query parameters [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] query_params = self.sanitize_for_serialization(query_params) [e2e-llm-inference-service] query_params = self.parameters_to_tuples(query_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] # post parameters [e2e-llm-inference-service] if post_params or files: [e2e-llm-inference-service] post_params = post_params if post_params else [] [e2e-llm-inference-service] post_params = self.sanitize_for_serialization(post_params) [e2e-llm-inference-service] post_params = self.parameters_to_tuples(post_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] post_params.extend(self.files_parameters(files)) [e2e-llm-inference-service] [e2e-llm-inference-service] # auth setting [e2e-llm-inference-service] self.update_params_for_auth(header_params, query_params, auth_settings) [e2e-llm-inference-service] [e2e-llm-inference-service] # body [e2e-llm-inference-service] if body: [e2e-llm-inference-service] body = self.sanitize_for_serialization(body) [e2e-llm-inference-service] [e2e-llm-inference-service] # request url [e2e-llm-inference-service] if _host is None: [e2e-llm-inference-service] url = self.configuration.host + resource_path [e2e-llm-inference-service] else: [e2e-llm-inference-service] # use server/host defined in path or operation instead [e2e-llm-inference-service] url = _host + resource_path [e2e-llm-inference-service] [e2e-llm-inference-service] # perform request and return response [e2e-llm-inference-service] > response_data = self.request( [e2e-llm-inference-service] method, url, query_params=query_params, headers=header_params, [e2e-llm-inference-service] post_params=post_params, body=body, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:180: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = 'https://afb35e615998e4024955f70c61cd36d7-c1814657712f161b.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-with-lora-adapters-3cbf5e9c/llminferenceserviceconfigs' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] post_params = [] [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-no-scheduler-lora-single-3cece2ef', 'namespace': 'e2e-test-llm-with-lora-adapters-3cbf5e9c'}, 'spec': {'router': {'route': {}}}} [e2e-llm-inference-service] _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] post_params=None, body=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Makes the HTTP request using RESTClient.""" [e2e-llm-inference-service] if method == "GET": [e2e-llm-inference-service] return self.rest_client.GET(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif method == "HEAD": [e2e-llm-inference-service] return self.rest_client.HEAD(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif method == "OPTIONS": [e2e-llm-inference-service] return self.rest_client.OPTIONS(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout) [e2e-llm-inference-service] elif method == "POST": [e2e-llm-inference-service] > return self.rest_client.POST(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] post_params=post_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:391: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] url = 'https://afb35e615998e4024955f70c61cd36d7-c1814657712f161b.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-with-lora-adapters-3cbf5e9c/llminferenceserviceconfigs' [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] query_params = [], post_params = [] [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-no-scheduler-lora-single-3cece2ef', 'namespace': 'e2e-test-llm-with-lora-adapters-3cbf5e9c'}, 'spec': {'router': {'route': {}}}} [e2e-llm-inference-service] _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def POST(self, url, headers=None, query_params=None, post_params=None, [e2e-llm-inference-service] body=None, _preload_content=True, _request_timeout=None): [e2e-llm-inference-service] > return self.request("POST", url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] post_params=post_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] body=body) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:279: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = 'https://afb35e615998e4024955f70c61cd36d7-c1814657712f161b.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-with-lora-adapters-3cbf5e9c/llminferenceserviceconfigs' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-no-scheduler-lora-single-3cece2ef', 'namespace': 'e2e-test-llm-with-lora-adapters-3cbf5e9c'}, 'spec': {'router': {'route': {}}}} [e2e-llm-inference-service] post_params = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] body=None, post_params=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Perform requests. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: http request method [e2e-llm-inference-service] :param url: http request url [e2e-llm-inference-service] :param query_params: query parameters in the url [e2e-llm-inference-service] :param headers: http request headers [e2e-llm-inference-service] :param body: request json body, for `application/json` [e2e-llm-inference-service] :param post_params: request post parameters, [e2e-llm-inference-service] `application/x-www-form-urlencoded` [e2e-llm-inference-service] and `multipart/form-data` [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] """ [e2e-llm-inference-service] method = method.upper() [e2e-llm-inference-service] assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT', [e2e-llm-inference-service] 'PATCH', 'OPTIONS'] [e2e-llm-inference-service] [e2e-llm-inference-service] if post_params and body: [e2e-llm-inference-service] raise ApiValueError( [e2e-llm-inference-service] "body parameter cannot be used with post_params parameter." [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] post_params = post_params or {} [e2e-llm-inference-service] headers = headers or {} [e2e-llm-inference-service] [e2e-llm-inference-service] timeout = None [e2e-llm-inference-service] if _request_timeout: [e2e-llm-inference-service] if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)): # noqa: E501,F821 [e2e-llm-inference-service] timeout = urllib3.Timeout(total=_request_timeout) [e2e-llm-inference-service] elif (isinstance(_request_timeout, tuple) and [e2e-llm-inference-service] len(_request_timeout) == 2): [e2e-llm-inference-service] timeout = urllib3.Timeout( [e2e-llm-inference-service] connect=_request_timeout[0], read=_request_timeout[1]) [e2e-llm-inference-service] [e2e-llm-inference-service] if 'Content-Type' not in headers: [e2e-llm-inference-service] headers['Content-Type'] = 'application/json' [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE` [e2e-llm-inference-service] if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']: [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] url += '?' + urlencode(query_params) [e2e-llm-inference-service] if (re.search('json', headers['Content-Type'], re.IGNORECASE) or [e2e-llm-inference-service] headers['Content-Type'] == 'application/apply-patch+yaml'): [e2e-llm-inference-service] if headers['Content-Type'] == 'application/json-patch+json': [e2e-llm-inference-service] if not isinstance(body, list): [e2e-llm-inference-service] headers['Content-Type'] = \ [e2e-llm-inference-service] 'application/strategic-merge-patch+json' [e2e-llm-inference-service] request_body = None [e2e-llm-inference-service] if body is not None: [e2e-llm-inference-service] request_body = json.dumps(body) [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'application/x-www-form-urlencoded': # noqa: E501 [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=False, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'multipart/form-data': [e2e-llm-inference-service] # must del headers['Content-Type'], or the correct [e2e-llm-inference-service] # Content-Type which generated by urllib3 will be [e2e-llm-inference-service] # overwritten. [e2e-llm-inference-service] del headers['Content-Type'] [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=True, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] # Pass a `string` parameter directly in the body to support [e2e-llm-inference-service] # other content types than Json when `body` argument is [e2e-llm-inference-service] # provided in serialized form [e2e-llm-inference-service] elif isinstance(body, str) or isinstance(body, bytes): [e2e-llm-inference-service] request_body = body [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] else: [e2e-llm-inference-service] # Cannot generate the request from given parameters [e2e-llm-inference-service] msg = """Cannot prepare a request message for provided [e2e-llm-inference-service] arguments. Please check that your arguments match [e2e-llm-inference-service] declared content type.""" [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] # For `GET`, `HEAD` [e2e-llm-inference-service] else: [e2e-llm-inference-service] r = self.pool_manager.request(method, url, [e2e-llm-inference-service] fields=query_params, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] except urllib3.exceptions.SSLError as e: [e2e-llm-inference-service] msg = "{0}\n{1}".format(type(e).__name__, str(e)) [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] [e2e-llm-inference-service] if _preload_content: [e2e-llm-inference-service] r = RESTResponse(r) [e2e-llm-inference-service] [e2e-llm-inference-service] # In the python 3, the response.data is bytes. [e2e-llm-inference-service] # we need to decode it to string. [e2e-llm-inference-service] if six.PY3: [e2e-llm-inference-service] r.data = r.data.decode('utf8') [e2e-llm-inference-service] [e2e-llm-inference-service] # log response body [e2e-llm-inference-service] logger.debug("response body: %s", r.data) [e2e-llm-inference-service] [e2e-llm-inference-service] if not 200 <= r.status <= 299: [e2e-llm-inference-service] > raise ApiException(http_resp=r) [e2e-llm-inference-service] E kubernetes.client.exceptions.ApiException: (500) [e2e-llm-inference-service] E Reason: Internal Server Error [e2e-llm-inference-service] E HTTP response headers: HTTPHeaderDict({'Audit-Id': '83bd366d-8dce-41be-bae9-d30180b5b35e', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'bf546a5c-3f60-4261-9b3f-c3b9e58fb8c8', 'X-Kubernetes-Pf-Prioritylevel-Uid': '39b8073e-7e81-4403-a6ca-a29718cbfcdd', 'Date': 'Mon, 27 Jul 2026 18:12:42 GMT', 'Content-Length': '833'}) [e2e-llm-inference-service] E HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha1.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/validate-serving-kserve-io-v1alpha1-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha1.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/validate-serving-kserve-io-v1alpha1-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\""}]},"code":500} [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:238: ApiException [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-with-lora-adapters-3cbf5e9c [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-with-lora-adapters-3cbf5e9c [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-with-lora-adapters-3cbf5e9c [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-with-lora-adapters-3cbf5e9c with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-with-lora-adapters-3cbf5e9c [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-with-lora-adapters-3cbf5e9c [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_with_lora_adapters] [2026-07-27T18:12:42.292491] start - args=(), kwargs={'test_case': LoRATestCase(base_refs=['router-no-scheduler', 'workload-single-cpu', 'model-fb-opt-125m-with-lora-hf'], prompt='What is Kubernetes?', expected_adapter_names=['lora-adapter-1'], service_name='lora-single-adapter-test', endpoint='/v1/completions', max_tokens=100, wait_timeout=900, response_timeout=60), 'test_namespace': 'e2e-test-llm-with-lora-adapters-3cbf5e9c'} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_lora_adapters:test_llm_lora_adapters.py:118 Creating LLMInferenceServiceConfig: router-no-scheduler-lora-single-3cece2ef [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-no-scheduler-lora-single-3cece2ef in namespace e2e-test-llm-with-lora-adapters-3cbf5e9c [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-no-scheduler-lora-single-3cece2ef [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:243 # Diagnostics for 'lora-single-adapter-test' in 'e2e-test-llm-with-lora-adapters-3cbf5e9c' [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:244 --- [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:247 # LLMInferenceService lora-single-adapter-test [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:258 # failed to dump LLMInferenceService: (404) [e2e-llm-inference-service] Reason: Not Found [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': 'de285d37-3bef-41ac-8083-f1ee1940a290', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'bf546a5c-3f60-4261-9b3f-c3b9e58fb8c8', 'X-Kubernetes-Pf-Prioritylevel-Uid': '39b8073e-7e81-4403-a6ca-a29718cbfcdd', 'Date': 'Mon, 27 Jul 2026 18:12:42 GMT', 'Content-Length': '294'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"llminferenceservices.serving.kserve.io \"lora-single-adapter-test\" not found","reason":"NotFound","details":{"name":"lora-single-adapter-test","group":"serving.kserve.io","kind":"llminferenceservices"},"code":404} [e2e-llm-inference-service] [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:64 ℹ️ # No events found in namespace e2e-test-llm-with-lora-adapters-3cbf5e9c [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:166 # no pods found in e2e-test-llm-with-lora-adapters-3cbf5e9c matching app.kubernetes.io/part-of=llminferenceservice,app.kubernetes.io/name=lora-single-adapter-test [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_with_lora_adapters] [2026-07-27T18:12:43.184408] end - ❌ 0.892s: (500) [e2e-llm-inference-service] Reason: Internal Server Error [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '83bd366d-8dce-41be-bae9-d30180b5b35e', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'bf546a5c-3f60-4261-9b3f-c3b9e58fb8c8', 'X-Kubernetes-Pf-Prioritylevel-Uid': '39b8073e-7e81-4403-a6ca-a29718cbfcdd', 'Date': 'Mon, 27 Jul 2026 18:12:42 GMT', 'Content-Length': '833'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha1.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/validate-serving-kserve-io-v1alpha1-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha1.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/validate-serving-kserve-io-v1alpha1-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\""}]},"code":500} [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-with-lora-adapters-3cbf5e9c (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _____________ test_llm_with_lora_adapters[multiple-lora-adapters] ______________ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] llm_config = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-no-scheduler-lora-multip-343dff1f', 'namespace': 'e2e-test-llm-with-lora-adapters-bbeb2681'}, 'spec': {'router': {'route': {}}}} [e2e-llm-inference-service] namespace = 'e2e-test-llm-with-lora-adapters-bbeb2681' [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_or_update_llmisvc_config(kserve_client, llm_config, namespace=None): [e2e-llm-inference-service] """Create or update an LLMInferenceServiceConfig resource.""" [e2e-llm-inference-service] version = llm_config["apiVersion"].split("/")[1] [e2e-llm-inference-service] [e2e-llm-inference-service] if namespace is None: [e2e-llm-inference-service] namespace = llm_config.get("metadata", {}).get("namespace", "default") [e2e-llm-inference-service] [e2e-llm-inference-service] name = llm_config.get("metadata", {}).get("name") [e2e-llm-inference-service] if not name: [e2e-llm-inference-service] raise ValueError("LLMInferenceServiceConfig must have a name in metadata") [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Checking LLMInferenceServiceConfig {name} in namespace {namespace}") [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > existing_config = kserve_client.api_instance.get_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICECONFIG, [e2e-llm-inference-service] name, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/fixtures.py:1721: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-with-lora-adapters-bbeb2681' [e2e-llm-inference-service] plural = 'llminferenceserviceconfigs' [e2e-llm-inference-service] name = 'router-no-scheduler-lora-multip-343dff1f' [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] [e2e-llm-inference-service] def get_namespaced_custom_object(self, group, version, namespace, plural, name, **kwargs): # noqa: E501 [e2e-llm-inference-service] """get_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Returns a namespace scoped custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.get_namespaced_custom_object(group, version, namespace, plural, name, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: the custom resource's group (required) [e2e-llm-inference-service] :param str version: the custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: the custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param str name: the custom object's name (required) [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: object [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] kwargs['_return_http_data_only'] = True [e2e-llm-inference-service] > return self.get_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, **kwargs) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:1632: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-with-lora-adapters-bbeb2681' [e2e-llm-inference-service] plural = 'llminferenceserviceconfigs' [e2e-llm-inference-service] name = 'router-no-scheduler-lora-multip-343dff1f' [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] local_var_params = {'_return_http_data_only': True, 'all_params': ['group', 'version', 'namespace', 'plural', 'name', 'async_req', ...], 'auth_settings': ['BearerToken'], 'body_params': None, ...} [e2e-llm-inference-service] all_params = ['group', 'version', 'namespace', 'plural', 'name', 'async_req', ...] [e2e-llm-inference-service] key = '_return_http_data_only', val = True, collection_formats = {} [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'name': 'router-no-scheduler-lora-multip-343dff1f', 'namespace': 'e2e-test-llm-with-lora-adapters-bbeb2681', 'plural': 'llminferenceserviceconfigs', ...} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] [e2e-llm-inference-service] def get_namespaced_custom_object_with_http_info(self, group, version, namespace, plural, name, **kwargs): # noqa: E501 [e2e-llm-inference-service] """get_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Returns a namespace scoped custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.get_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: the custom resource's group (required) [e2e-llm-inference-service] :param str version: the custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: the custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param str name: the custom object's name (required) [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: tuple(object, status_code(int), headers(HTTPHeaderDict)) [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] local_var_params = locals() [e2e-llm-inference-service] [e2e-llm-inference-service] all_params = [ [e2e-llm-inference-service] 'group', [e2e-llm-inference-service] 'version', [e2e-llm-inference-service] 'namespace', [e2e-llm-inference-service] 'plural', [e2e-llm-inference-service] 'name' [e2e-llm-inference-service] ] [e2e-llm-inference-service] all_params.extend( [e2e-llm-inference-service] [ [e2e-llm-inference-service] 'async_req', [e2e-llm-inference-service] '_return_http_data_only', [e2e-llm-inference-service] '_preload_content', [e2e-llm-inference-service] '_request_timeout' [e2e-llm-inference-service] ] [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] for key, val in six.iteritems(local_var_params['kwargs']): [e2e-llm-inference-service] if key not in all_params: [e2e-llm-inference-service] raise ApiTypeError( [e2e-llm-inference-service] "Got an unexpected keyword argument '%s'" [e2e-llm-inference-service] " to method get_namespaced_custom_object" % key [e2e-llm-inference-service] ) [e2e-llm-inference-service] local_var_params[key] = val [e2e-llm-inference-service] del local_var_params['kwargs'] [e2e-llm-inference-service] # verify the required parameter 'group' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('group' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['group'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `group` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'version' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('version' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['version'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `version` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'namespace' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('namespace' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['namespace'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `namespace` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'plural' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('plural' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['plural'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `plural` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'name' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('name' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['name'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `name` when calling `get_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] collection_formats = {} [e2e-llm-inference-service] [e2e-llm-inference-service] path_params = {} [e2e-llm-inference-service] if 'group' in local_var_params: [e2e-llm-inference-service] path_params['group'] = local_var_params['group'] # noqa: E501 [e2e-llm-inference-service] if 'version' in local_var_params: [e2e-llm-inference-service] path_params['version'] = local_var_params['version'] # noqa: E501 [e2e-llm-inference-service] if 'namespace' in local_var_params: [e2e-llm-inference-service] path_params['namespace'] = local_var_params['namespace'] # noqa: E501 [e2e-llm-inference-service] if 'plural' in local_var_params: [e2e-llm-inference-service] path_params['plural'] = local_var_params['plural'] # noqa: E501 [e2e-llm-inference-service] if 'name' in local_var_params: [e2e-llm-inference-service] path_params['name'] = local_var_params['name'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] [e2e-llm-inference-service] header_params = {} [e2e-llm-inference-service] [e2e-llm-inference-service] form_params = [] [e2e-llm-inference-service] local_var_files = {} [e2e-llm-inference-service] [e2e-llm-inference-service] body_params = None [e2e-llm-inference-service] # HTTP header `Accept` [e2e-llm-inference-service] header_params['Accept'] = self.api_client.select_header_accept( [e2e-llm-inference-service] ['application/json']) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] # Authentication setting [e2e-llm-inference-service] auth_settings = ['BearerToken'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] > return self.api_client.call_api( [e2e-llm-inference-service] '/apis/{group}/{version}/namespaces/{namespace}/{plural}/{name}', 'GET', [e2e-llm-inference-service] path_params, [e2e-llm-inference-service] query_params, [e2e-llm-inference-service] header_params, [e2e-llm-inference-service] body=body_params, [e2e-llm-inference-service] post_params=form_params, [e2e-llm-inference-service] files=local_var_files, [e2e-llm-inference-service] response_type='object', # noqa: E501 [e2e-llm-inference-service] auth_settings=auth_settings, [e2e-llm-inference-service] async_req=local_var_params.get('async_req'), [e2e-llm-inference-service] _return_http_data_only=local_var_params.get('_return_http_data_only'), # noqa: E501 [e2e-llm-inference-service] _preload_content=local_var_params.get('_preload_content', True), [e2e-llm-inference-service] _request_timeout=local_var_params.get('_request_timeout'), [e2e-llm-inference-service] collection_formats=collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:1739: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/{group}/{version}/namespaces/{namespace}/{plural}/{name}' [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'name': 'router-no-scheduler-lora-multip-343dff1f', 'namespace': 'e2e-test-llm-with-lora-adapters-bbeb2681', 'plural': 'llminferenceserviceconfigs', ...} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], async_req = None, _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def call_api(self, resource_path, method, [e2e-llm-inference-service] path_params=None, query_params=None, header_params=None, [e2e-llm-inference-service] body=None, post_params=None, files=None, [e2e-llm-inference-service] response_type=None, auth_settings=None, async_req=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] """Makes the HTTP request (synchronous) and returns deserialized data. [e2e-llm-inference-service] [e2e-llm-inference-service] To make an async_req request, set the async_req parameter. [e2e-llm-inference-service] [e2e-llm-inference-service] :param resource_path: Path to method endpoint. [e2e-llm-inference-service] :param method: Method to call. [e2e-llm-inference-service] :param path_params: Path parameters in the url. [e2e-llm-inference-service] :param query_params: Query parameters in the url. [e2e-llm-inference-service] :param header_params: Header parameters to be [e2e-llm-inference-service] placed in the request header. [e2e-llm-inference-service] :param body: Request body. [e2e-llm-inference-service] :param post_params dict: Request post form parameters, [e2e-llm-inference-service] for `application/x-www-form-urlencoded`, `multipart/form-data`. [e2e-llm-inference-service] :param auth_settings list: Auth Settings names for the request. [e2e-llm-inference-service] :param response: Response data type. [e2e-llm-inference-service] :param files dict: key -> filename, value -> filepath, [e2e-llm-inference-service] for `multipart/form-data`. [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param collection_formats: dict of collection formats for path, query, [e2e-llm-inference-service] header, and post parameters. [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: [e2e-llm-inference-service] If async_req parameter is True, [e2e-llm-inference-service] the request will be called asynchronously. [e2e-llm-inference-service] The method will return the request thread. [e2e-llm-inference-service] If parameter async_req is False or missing, [e2e-llm-inference-service] then the method will return the response directly. [e2e-llm-inference-service] """ [e2e-llm-inference-service] if not async_req: [e2e-llm-inference-service] > return self.__call_api(resource_path, method, [e2e-llm-inference-service] path_params, query_params, header_params, [e2e-llm-inference-service] body, post_params, files, [e2e-llm-inference-service] response_type, auth_settings, [e2e-llm-inference-service] _return_http_data_only, collection_formats, [e2e-llm-inference-service] _preload_content, _request_timeout, _host) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:348: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-with-lora-adapters-bbeb2681/llminferenceserviceconfigs/router-no-scheduler-lora-multip-343dff1f' [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] path_params = [('group', 'serving.kserve.io'), ('version', 'v1alpha1'), ('namespace', 'e2e-test-llm-with-lora-adapters-bbeb2681'), ('plural', 'llminferenceserviceconfigs'), ('name', 'router-no-scheduler-lora-multip-343dff1f')] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def __call_api( [e2e-llm-inference-service] self, resource_path, method, path_params=None, [e2e-llm-inference-service] query_params=None, header_params=None, body=None, post_params=None, [e2e-llm-inference-service] files=None, response_type=None, auth_settings=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] [e2e-llm-inference-service] config = self.configuration [e2e-llm-inference-service] [e2e-llm-inference-service] # header parameters [e2e-llm-inference-service] header_params = header_params or {} [e2e-llm-inference-service] header_params.update(self.default_headers) [e2e-llm-inference-service] if self.cookie: [e2e-llm-inference-service] header_params['Cookie'] = self.cookie [e2e-llm-inference-service] if header_params: [e2e-llm-inference-service] header_params = self.sanitize_for_serialization(header_params) [e2e-llm-inference-service] header_params = dict(self.parameters_to_tuples(header_params, [e2e-llm-inference-service] collection_formats)) [e2e-llm-inference-service] [e2e-llm-inference-service] # path parameters [e2e-llm-inference-service] if path_params: [e2e-llm-inference-service] path_params = self.sanitize_for_serialization(path_params) [e2e-llm-inference-service] path_params = self.parameters_to_tuples(path_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] for k, v in path_params: [e2e-llm-inference-service] # specified safe chars, encode everything [e2e-llm-inference-service] resource_path = resource_path.replace( [e2e-llm-inference-service] '{%s}' % k, [e2e-llm-inference-service] quote(str(v), safe=config.safe_chars_for_path_param) [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # query parameters [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] query_params = self.sanitize_for_serialization(query_params) [e2e-llm-inference-service] query_params = self.parameters_to_tuples(query_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] # post parameters [e2e-llm-inference-service] if post_params or files: [e2e-llm-inference-service] post_params = post_params if post_params else [] [e2e-llm-inference-service] post_params = self.sanitize_for_serialization(post_params) [e2e-llm-inference-service] post_params = self.parameters_to_tuples(post_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] post_params.extend(self.files_parameters(files)) [e2e-llm-inference-service] [e2e-llm-inference-service] # auth setting [e2e-llm-inference-service] self.update_params_for_auth(header_params, query_params, auth_settings) [e2e-llm-inference-service] [e2e-llm-inference-service] # body [e2e-llm-inference-service] if body: [e2e-llm-inference-service] body = self.sanitize_for_serialization(body) [e2e-llm-inference-service] [e2e-llm-inference-service] # request url [e2e-llm-inference-service] if _host is None: [e2e-llm-inference-service] url = self.configuration.host + resource_path [e2e-llm-inference-service] else: [e2e-llm-inference-service] # use server/host defined in path or operation instead [e2e-llm-inference-service] url = _host + resource_path [e2e-llm-inference-service] [e2e-llm-inference-service] # perform request and return response [e2e-llm-inference-service] > response_data = self.request( [e2e-llm-inference-service] method, url, query_params=query_params, headers=header_params, [e2e-llm-inference-service] post_params=post_params, body=body, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:180: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] url = 'https://afb35e615998e4024955f70c61cd36d7-c1814657712f161b.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1a...amespaces/e2e-test-llm-with-lora-adapters-bbeb2681/llminferenceserviceconfigs/router-no-scheduler-lora-multip-343dff1f' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] post_params = [], body = None, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] post_params=None, body=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Makes the HTTP request using RESTClient.""" [e2e-llm-inference-service] if method == "GET": [e2e-llm-inference-service] > return self.rest_client.GET(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:373: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] url = 'https://afb35e615998e4024955f70c61cd36d7-c1814657712f161b.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1a...amespaces/e2e-test-llm-with-lora-adapters-bbeb2681/llminferenceserviceconfigs/router-no-scheduler-lora-multip-343dff1f' [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] query_params = [], _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def GET(self, url, headers=None, query_params=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] > return self.request("GET", url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] query_params=query_params) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:244: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'GET' [e2e-llm-inference-service] url = 'https://afb35e615998e4024955f70c61cd36d7-c1814657712f161b.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1a...amespaces/e2e-test-llm-with-lora-adapters-bbeb2681/llminferenceserviceconfigs/router-no-scheduler-lora-multip-343dff1f' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = None, post_params = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] body=None, post_params=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Perform requests. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: http request method [e2e-llm-inference-service] :param url: http request url [e2e-llm-inference-service] :param query_params: query parameters in the url [e2e-llm-inference-service] :param headers: http request headers [e2e-llm-inference-service] :param body: request json body, for `application/json` [e2e-llm-inference-service] :param post_params: request post parameters, [e2e-llm-inference-service] `application/x-www-form-urlencoded` [e2e-llm-inference-service] and `multipart/form-data` [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] """ [e2e-llm-inference-service] method = method.upper() [e2e-llm-inference-service] assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT', [e2e-llm-inference-service] 'PATCH', 'OPTIONS'] [e2e-llm-inference-service] [e2e-llm-inference-service] if post_params and body: [e2e-llm-inference-service] raise ApiValueError( [e2e-llm-inference-service] "body parameter cannot be used with post_params parameter." [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] post_params = post_params or {} [e2e-llm-inference-service] headers = headers or {} [e2e-llm-inference-service] [e2e-llm-inference-service] timeout = None [e2e-llm-inference-service] if _request_timeout: [e2e-llm-inference-service] if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)): # noqa: E501,F821 [e2e-llm-inference-service] timeout = urllib3.Timeout(total=_request_timeout) [e2e-llm-inference-service] elif (isinstance(_request_timeout, tuple) and [e2e-llm-inference-service] len(_request_timeout) == 2): [e2e-llm-inference-service] timeout = urllib3.Timeout( [e2e-llm-inference-service] connect=_request_timeout[0], read=_request_timeout[1]) [e2e-llm-inference-service] [e2e-llm-inference-service] if 'Content-Type' not in headers: [e2e-llm-inference-service] headers['Content-Type'] = 'application/json' [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE` [e2e-llm-inference-service] if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']: [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] url += '?' + urlencode(query_params) [e2e-llm-inference-service] if (re.search('json', headers['Content-Type'], re.IGNORECASE) or [e2e-llm-inference-service] headers['Content-Type'] == 'application/apply-patch+yaml'): [e2e-llm-inference-service] if headers['Content-Type'] == 'application/json-patch+json': [e2e-llm-inference-service] if not isinstance(body, list): [e2e-llm-inference-service] headers['Content-Type'] = \ [e2e-llm-inference-service] 'application/strategic-merge-patch+json' [e2e-llm-inference-service] request_body = None [e2e-llm-inference-service] if body is not None: [e2e-llm-inference-service] request_body = json.dumps(body) [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'application/x-www-form-urlencoded': # noqa: E501 [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=False, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'multipart/form-data': [e2e-llm-inference-service] # must del headers['Content-Type'], or the correct [e2e-llm-inference-service] # Content-Type which generated by urllib3 will be [e2e-llm-inference-service] # overwritten. [e2e-llm-inference-service] del headers['Content-Type'] [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=True, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] # Pass a `string` parameter directly in the body to support [e2e-llm-inference-service] # other content types than Json when `body` argument is [e2e-llm-inference-service] # provided in serialized form [e2e-llm-inference-service] elif isinstance(body, str) or isinstance(body, bytes): [e2e-llm-inference-service] request_body = body [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] else: [e2e-llm-inference-service] # Cannot generate the request from given parameters [e2e-llm-inference-service] msg = """Cannot prepare a request message for provided [e2e-llm-inference-service] arguments. Please check that your arguments match [e2e-llm-inference-service] declared content type.""" [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] # For `GET`, `HEAD` [e2e-llm-inference-service] else: [e2e-llm-inference-service] r = self.pool_manager.request(method, url, [e2e-llm-inference-service] fields=query_params, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] except urllib3.exceptions.SSLError as e: [e2e-llm-inference-service] msg = "{0}\n{1}".format(type(e).__name__, str(e)) [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] [e2e-llm-inference-service] if _preload_content: [e2e-llm-inference-service] r = RESTResponse(r) [e2e-llm-inference-service] [e2e-llm-inference-service] # In the python 3, the response.data is bytes. [e2e-llm-inference-service] # we need to decode it to string. [e2e-llm-inference-service] if six.PY3: [e2e-llm-inference-service] r.data = r.data.decode('utf8') [e2e-llm-inference-service] [e2e-llm-inference-service] # log response body [e2e-llm-inference-service] logger.debug("response body: %s", r.data) [e2e-llm-inference-service] [e2e-llm-inference-service] if not 200 <= r.status <= 299: [e2e-llm-inference-service] > raise ApiException(http_resp=r) [e2e-llm-inference-service] E kubernetes.client.exceptions.ApiException: (404) [e2e-llm-inference-service] E Reason: Not Found [e2e-llm-inference-service] E HTTP response headers: HTTPHeaderDict({'Audit-Id': 'fe59c0d3-f3f4-44a3-8d60-d0bbdc2e323a', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'bf546a5c-3f60-4261-9b3f-c3b9e58fb8c8', 'X-Kubernetes-Pf-Prioritylevel-Uid': '39b8073e-7e81-4403-a6ca-a29718cbfcdd', 'Date': 'Mon, 27 Jul 2026 18:12:43 GMT', 'Content-Length': '338'}) [e2e-llm-inference-service] E HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"llminferenceserviceconfigs.serving.kserve.io \"router-no-scheduler-lora-multip-343dff1f\" not found","reason":"NotFound","details":{"name":"router-no-scheduler-lora-multip-343dff1f","group":"serving.kserve.io","kind":"llminferenceserviceconfigs"},"code":404} [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:238: ApiException [e2e-llm-inference-service] [e2e-llm-inference-service] During handling of the above exception, another exception occurred: [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = LoRATestCase(base_refs=['router-no-scheduler', 'workload-single-cpu', 'model-fb-opt-125m-with-multiple-lora'], prompt=..._name='lora-multiple-adapters-test', endpoint='/v1/completions', max_tokens=100, wait_timeout=900, response_timeout=60) [e2e-llm-inference-service] test_namespace = 'e2e-test-llm-with-lora-adapters-bbeb2681' [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] LoRATestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is Kubernetes?", [e2e-llm-inference-service] expected_adapter_names=["lora-adapter-1"], [e2e-llm-inference-service] service_name="lora-single-adapter-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] id="single-lora-adapter-hf", [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] LoRATestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-multiple-lora", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Explain machine learning in simple terms.", [e2e-llm-inference-service] expected_adapter_names=["lora-adapter-1", "lora-adapter-2"], [e2e-llm-inference-service] service_name="lora-multiple-adapters-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] id="multiple-lora-adapters", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_with_lora_adapters(test_case: LoRATestCase, test_namespace): [e2e-llm-inference-service] """Test LLMInferenceService with LoRA adapters.""" [e2e-llm-inference-service] > run_lora_test(test_case, namespace=test_namespace) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py:252: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = LoRATestCase(base_refs=['router-no-scheduler', 'workload-single-cpu', 'model-fb-opt-125m-with-multiple-lora'], prompt=..._name='lora-multiple-adapters-test', endpoint='/v1/completions', max_tokens=100, wait_timeout=900, response_timeout=60) [e2e-llm-inference-service] namespace = 'e2e-test-llm-with-lora-adapters-bbeb2681' [e2e-llm-inference-service] [e2e-llm-inference-service] def run_lora_test(test_case: LoRATestCase, namespace: str): [e2e-llm-inference-service] """Execute a LoRA adapter test case.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] test_id = generate_test_id(test_case) [e2e-llm-inference-service] service_name = test_case.service_name or f"lora-test-{test_id}" [e2e-llm-inference-service] model_name = _get_model_name_from_configs(test_case.base_refs) [e2e-llm-inference-service] created_configs = [] [e2e-llm-inference-service] llm_service = None [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Create unique LLMInferenceServiceConfig resources for each base ref [e2e-llm-inference-service] unique_base_refs = [] [e2e-llm-inference-service] for base_ref in test_case.base_refs: [e2e-llm-inference-service] if base_ref not in LLMINFERENCESERVICE_CONFIGS: [e2e-llm-inference-service] raise ValueError(f"Unknown base reference: {base_ref}") [e2e-llm-inference-service] [e2e-llm-inference-service] # Generate unique config name to avoid conflicts in parallel test runs [e2e-llm-inference-service] unique_config_name = generate_k8s_safe_suffix(base_ref, [service_name]) [e2e-llm-inference-service] unique_base_refs.append(unique_config_name) [e2e-llm-inference-service] [e2e-llm-inference-service] config = LLMINFERENCESERVICE_CONFIGS[base_ref] [e2e-llm-inference-service] config_body = { [e2e-llm-inference-service] "apiVersion": "serving.kserve.io/v1alpha1", [e2e-llm-inference-service] "kind": "LLMInferenceServiceConfig", [e2e-llm-inference-service] "metadata": { [e2e-llm-inference-service] "name": unique_config_name, [e2e-llm-inference-service] "namespace": namespace, [e2e-llm-inference-service] }, [e2e-llm-inference-service] "spec": config, [e2e-llm-inference-service] } [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] "Creating LLMInferenceServiceConfig: %s", [e2e-llm-inference-service] unique_config_name, [e2e-llm-inference-service] ) [e2e-llm-inference-service] > _create_or_update_llmisvc_config(kserve_client, config_body, namespace) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py:122: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] llm_config = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-no-scheduler-lora-multip-343dff1f', 'namespace': 'e2e-test-llm-with-lora-adapters-bbeb2681'}, 'spec': {'router': {'route': {}}}} [e2e-llm-inference-service] namespace = 'e2e-test-llm-with-lora-adapters-bbeb2681' [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_or_update_llmisvc_config(kserve_client, llm_config, namespace=None): [e2e-llm-inference-service] """Create or update an LLMInferenceServiceConfig resource.""" [e2e-llm-inference-service] version = llm_config["apiVersion"].split("/")[1] [e2e-llm-inference-service] [e2e-llm-inference-service] if namespace is None: [e2e-llm-inference-service] namespace = llm_config.get("metadata", {}).get("namespace", "default") [e2e-llm-inference-service] [e2e-llm-inference-service] name = llm_config.get("metadata", {}).get("name") [e2e-llm-inference-service] if not name: [e2e-llm-inference-service] raise ValueError("LLMInferenceServiceConfig must have a name in metadata") [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Checking LLMInferenceServiceConfig {name} in namespace {namespace}") [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] existing_config = kserve_client.api_instance.get_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICECONFIG, [e2e-llm-inference-service] name, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llm_config["metadata"] = existing_config["metadata"] [e2e-llm-inference-service] [e2e-llm-inference-service] outputs = kserve_client.api_instance.replace_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICECONFIG, [e2e-llm-inference-service] name, [e2e-llm-inference-service] llm_config, [e2e-llm-inference-service] ) [e2e-llm-inference-service] logger.info(f"✓ Successfully updated LLMInferenceServiceConfig {name}") [e2e-llm-inference-service] return outputs [e2e-llm-inference-service] [e2e-llm-inference-service] except client.rest.ApiException as e: [e2e-llm-inference-service] if e.status == 404: # Not found - create it [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"Resource not found, creating LLMInferenceServiceConfig {name}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] > outputs = kserve_client.api_instance.create_namespaced_custom_object( [e2e-llm-inference-service] constants.KSERVE_GROUP, [e2e-llm-inference-service] version, [e2e-llm-inference-service] namespace, [e2e-llm-inference-service] KSERVE_PLURAL_LLMINFERENCESERVICECONFIG, [e2e-llm-inference-service] llm_config, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/fixtures.py:1747: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-with-lora-adapters-bbeb2681' [e2e-llm-inference-service] plural = 'llminferenceserviceconfigs' [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-no-scheduler-lora-multip-343dff1f', 'namespace': 'e2e-test-llm-with-lora-adapters-bbeb2681'}, 'spec': {'router': {'route': {}}}} [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] [e2e-llm-inference-service] def create_namespaced_custom_object(self, group, version, namespace, plural, body, **kwargs): # noqa: E501 [e2e-llm-inference-service] """create_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Creates a namespace scoped Custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.create_namespaced_custom_object(group, version, namespace, plural, body, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: The custom resource's group name (required) [e2e-llm-inference-service] :param str version: The custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: The custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param object body: The JSON schema of the Resource to create. (required) [e2e-llm-inference-service] :param str pretty: If 'true', then the output is pretty printed. [e2e-llm-inference-service] :param str dry_run: When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed [e2e-llm-inference-service] :param str field_manager: fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. [e2e-llm-inference-service] :param str field_validation: fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default in v1.23+ - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered. (optional) [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: object [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] kwargs['_return_http_data_only'] = True [e2e-llm-inference-service] > return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:231: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] group = 'serving.kserve.io', version = 'v1alpha1' [e2e-llm-inference-service] namespace = 'e2e-test-llm-with-lora-adapters-bbeb2681' [e2e-llm-inference-service] plural = 'llminferenceserviceconfigs' [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-no-scheduler-lora-multip-343dff1f', 'namespace': 'e2e-test-llm-with-lora-adapters-bbeb2681'}, 'spec': {'router': {'route': {}}}} [e2e-llm-inference-service] kwargs = {'_return_http_data_only': True} [e2e-llm-inference-service] local_var_params = {'_return_http_data_only': True, 'all_params': ['group', 'version', 'namespace', 'plural', 'body', 'pretty', ...], 'au...ra-multip-343dff1f', 'namespace': 'e2e-test-llm-with-lora-adapters-bbeb2681'}, 'spec': {'router': {'route': {}}}}, ...} [e2e-llm-inference-service] all_params = ['group', 'version', 'namespace', 'plural', 'body', 'pretty', ...] [e2e-llm-inference-service] key = '_return_http_data_only', val = True, collection_formats = {} [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'namespace': 'e2e-test-llm-with-lora-adapters-bbeb2681', 'plural': 'llminferenceserviceconfigs', 'version': 'v1alpha1'} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] [e2e-llm-inference-service] def create_namespaced_custom_object_with_http_info(self, group, version, namespace, plural, body, **kwargs): # noqa: E501 [e2e-llm-inference-service] """create_namespaced_custom_object # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] Creates a namespace scoped Custom object # noqa: E501 [e2e-llm-inference-service] This method makes a synchronous HTTP request by default. To make an [e2e-llm-inference-service] asynchronous HTTP request, please pass async_req=True [e2e-llm-inference-service] >>> thread = api.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, async_req=True) [e2e-llm-inference-service] >>> result = thread.get() [e2e-llm-inference-service] [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param str group: The custom resource's group name (required) [e2e-llm-inference-service] :param str version: The custom resource's version (required) [e2e-llm-inference-service] :param str namespace: The custom resource's namespace (required) [e2e-llm-inference-service] :param str plural: The custom resource's plural name. For TPRs this would be lowercase plural kind. (required) [e2e-llm-inference-service] :param object body: The JSON schema of the Resource to create. (required) [e2e-llm-inference-service] :param str pretty: If 'true', then the output is pretty printed. [e2e-llm-inference-service] :param str dry_run: When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed [e2e-llm-inference-service] :param str field_manager: fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. [e2e-llm-inference-service] :param str field_validation: fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default in v1.23+ - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered. (optional) [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: tuple(object, status_code(int), headers(HTTPHeaderDict)) [e2e-llm-inference-service] If the method is called asynchronously, [e2e-llm-inference-service] returns the request thread. [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] local_var_params = locals() [e2e-llm-inference-service] [e2e-llm-inference-service] all_params = [ [e2e-llm-inference-service] 'group', [e2e-llm-inference-service] 'version', [e2e-llm-inference-service] 'namespace', [e2e-llm-inference-service] 'plural', [e2e-llm-inference-service] 'body', [e2e-llm-inference-service] 'pretty', [e2e-llm-inference-service] 'dry_run', [e2e-llm-inference-service] 'field_manager', [e2e-llm-inference-service] 'field_validation' [e2e-llm-inference-service] ] [e2e-llm-inference-service] all_params.extend( [e2e-llm-inference-service] [ [e2e-llm-inference-service] 'async_req', [e2e-llm-inference-service] '_return_http_data_only', [e2e-llm-inference-service] '_preload_content', [e2e-llm-inference-service] '_request_timeout' [e2e-llm-inference-service] ] [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] for key, val in six.iteritems(local_var_params['kwargs']): [e2e-llm-inference-service] if key not in all_params: [e2e-llm-inference-service] raise ApiTypeError( [e2e-llm-inference-service] "Got an unexpected keyword argument '%s'" [e2e-llm-inference-service] " to method create_namespaced_custom_object" % key [e2e-llm-inference-service] ) [e2e-llm-inference-service] local_var_params[key] = val [e2e-llm-inference-service] del local_var_params['kwargs'] [e2e-llm-inference-service] # verify the required parameter 'group' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('group' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['group'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `group` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'version' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('version' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['version'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `version` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'namespace' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('namespace' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['namespace'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `namespace` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'plural' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('plural' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['plural'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `plural` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] # verify the required parameter 'body' is set [e2e-llm-inference-service] if self.api_client.client_side_validation and ('body' not in local_var_params or # noqa: E501 [e2e-llm-inference-service] local_var_params['body'] is None): # noqa: E501 [e2e-llm-inference-service] raise ApiValueError("Missing the required parameter `body` when calling `create_namespaced_custom_object`") # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] collection_formats = {} [e2e-llm-inference-service] [e2e-llm-inference-service] path_params = {} [e2e-llm-inference-service] if 'group' in local_var_params: [e2e-llm-inference-service] path_params['group'] = local_var_params['group'] # noqa: E501 [e2e-llm-inference-service] if 'version' in local_var_params: [e2e-llm-inference-service] path_params['version'] = local_var_params['version'] # noqa: E501 [e2e-llm-inference-service] if 'namespace' in local_var_params: [e2e-llm-inference-service] path_params['namespace'] = local_var_params['namespace'] # noqa: E501 [e2e-llm-inference-service] if 'plural' in local_var_params: [e2e-llm-inference-service] path_params['plural'] = local_var_params['plural'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] if 'pretty' in local_var_params and local_var_params['pretty'] is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('pretty', local_var_params['pretty'])) # noqa: E501 [e2e-llm-inference-service] if 'dry_run' in local_var_params and local_var_params['dry_run'] is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('dryRun', local_var_params['dry_run'])) # noqa: E501 [e2e-llm-inference-service] if 'field_manager' in local_var_params and local_var_params['field_manager'] is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('fieldManager', local_var_params['field_manager'])) # noqa: E501 [e2e-llm-inference-service] if 'field_validation' in local_var_params and local_var_params['field_validation'] is not None: # noqa: E501 [e2e-llm-inference-service] query_params.append(('fieldValidation', local_var_params['field_validation'])) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] header_params = {} [e2e-llm-inference-service] [e2e-llm-inference-service] form_params = [] [e2e-llm-inference-service] local_var_files = {} [e2e-llm-inference-service] [e2e-llm-inference-service] body_params = None [e2e-llm-inference-service] if 'body' in local_var_params: [e2e-llm-inference-service] body_params = local_var_params['body'] [e2e-llm-inference-service] # HTTP header `Accept` [e2e-llm-inference-service] header_params['Accept'] = self.api_client.select_header_accept( [e2e-llm-inference-service] ['application/json']) # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] # Authentication setting [e2e-llm-inference-service] auth_settings = ['BearerToken'] # noqa: E501 [e2e-llm-inference-service] [e2e-llm-inference-service] > return self.api_client.call_api( [e2e-llm-inference-service] '/apis/{group}/{version}/namespaces/{namespace}/{plural}', 'POST', [e2e-llm-inference-service] path_params, [e2e-llm-inference-service] query_params, [e2e-llm-inference-service] header_params, [e2e-llm-inference-service] body=body_params, [e2e-llm-inference-service] post_params=form_params, [e2e-llm-inference-service] files=local_var_files, [e2e-llm-inference-service] response_type='object', # noqa: E501 [e2e-llm-inference-service] auth_settings=auth_settings, [e2e-llm-inference-service] async_req=local_var_params.get('async_req'), [e2e-llm-inference-service] _return_http_data_only=local_var_params.get('_return_http_data_only'), # noqa: E501 [e2e-llm-inference-service] _preload_content=local_var_params.get('_preload_content', True), [e2e-llm-inference-service] _request_timeout=local_var_params.get('_request_timeout'), [e2e-llm-inference-service] collection_formats=collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api/custom_objects_api.py:354: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/{group}/{version}/namespaces/{namespace}/{plural}' [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] path_params = {'group': 'serving.kserve.io', 'namespace': 'e2e-test-llm-with-lora-adapters-bbeb2681', 'plural': 'llminferenceserviceconfigs', 'version': 'v1alpha1'} [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-no-scheduler-lora-multip-343dff1f', 'namespace': 'e2e-test-llm-with-lora-adapters-bbeb2681'}, 'spec': {'router': {'route': {}}}} [e2e-llm-inference-service] post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], async_req = None, _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def call_api(self, resource_path, method, [e2e-llm-inference-service] path_params=None, query_params=None, header_params=None, [e2e-llm-inference-service] body=None, post_params=None, files=None, [e2e-llm-inference-service] response_type=None, auth_settings=None, async_req=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] """Makes the HTTP request (synchronous) and returns deserialized data. [e2e-llm-inference-service] [e2e-llm-inference-service] To make an async_req request, set the async_req parameter. [e2e-llm-inference-service] [e2e-llm-inference-service] :param resource_path: Path to method endpoint. [e2e-llm-inference-service] :param method: Method to call. [e2e-llm-inference-service] :param path_params: Path parameters in the url. [e2e-llm-inference-service] :param query_params: Query parameters in the url. [e2e-llm-inference-service] :param header_params: Header parameters to be [e2e-llm-inference-service] placed in the request header. [e2e-llm-inference-service] :param body: Request body. [e2e-llm-inference-service] :param post_params dict: Request post form parameters, [e2e-llm-inference-service] for `application/x-www-form-urlencoded`, `multipart/form-data`. [e2e-llm-inference-service] :param auth_settings list: Auth Settings names for the request. [e2e-llm-inference-service] :param response: Response data type. [e2e-llm-inference-service] :param files dict: key -> filename, value -> filepath, [e2e-llm-inference-service] for `multipart/form-data`. [e2e-llm-inference-service] :param async_req bool: execute request asynchronously [e2e-llm-inference-service] :param _return_http_data_only: response data without head status code [e2e-llm-inference-service] and headers [e2e-llm-inference-service] :param collection_formats: dict of collection formats for path, query, [e2e-llm-inference-service] header, and post parameters. [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] :return: [e2e-llm-inference-service] If async_req parameter is True, [e2e-llm-inference-service] the request will be called asynchronously. [e2e-llm-inference-service] The method will return the request thread. [e2e-llm-inference-service] If parameter async_req is False or missing, [e2e-llm-inference-service] then the method will return the response directly. [e2e-llm-inference-service] """ [e2e-llm-inference-service] if not async_req: [e2e-llm-inference-service] > return self.__call_api(resource_path, method, [e2e-llm-inference-service] path_params, query_params, header_params, [e2e-llm-inference-service] body, post_params, files, [e2e-llm-inference-service] response_type, auth_settings, [e2e-llm-inference-service] _return_http_data_only, collection_formats, [e2e-llm-inference-service] _preload_content, _request_timeout, _host) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:348: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] resource_path = '/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-with-lora-adapters-bbeb2681/llminferenceserviceconfigs' [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] path_params = [('group', 'serving.kserve.io'), ('version', 'v1alpha1'), ('namespace', 'e2e-test-llm-with-lora-adapters-bbeb2681'), ('plural', 'llminferenceserviceconfigs')] [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] header_params = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-no-scheduler-lora-multip-343dff1f', 'namespace': 'e2e-test-llm-with-lora-adapters-bbeb2681'}, 'spec': {'router': {'route': {}}}} [e2e-llm-inference-service] post_params = [], files = {}, response_type = 'object' [e2e-llm-inference-service] auth_settings = ['BearerToken'], _return_http_data_only = True [e2e-llm-inference-service] collection_formats = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] _host = None [e2e-llm-inference-service] [e2e-llm-inference-service] def __call_api( [e2e-llm-inference-service] self, resource_path, method, path_params=None, [e2e-llm-inference-service] query_params=None, header_params=None, body=None, post_params=None, [e2e-llm-inference-service] files=None, response_type=None, auth_settings=None, [e2e-llm-inference-service] _return_http_data_only=None, collection_formats=None, [e2e-llm-inference-service] _preload_content=True, _request_timeout=None, _host=None): [e2e-llm-inference-service] [e2e-llm-inference-service] config = self.configuration [e2e-llm-inference-service] [e2e-llm-inference-service] # header parameters [e2e-llm-inference-service] header_params = header_params or {} [e2e-llm-inference-service] header_params.update(self.default_headers) [e2e-llm-inference-service] if self.cookie: [e2e-llm-inference-service] header_params['Cookie'] = self.cookie [e2e-llm-inference-service] if header_params: [e2e-llm-inference-service] header_params = self.sanitize_for_serialization(header_params) [e2e-llm-inference-service] header_params = dict(self.parameters_to_tuples(header_params, [e2e-llm-inference-service] collection_formats)) [e2e-llm-inference-service] [e2e-llm-inference-service] # path parameters [e2e-llm-inference-service] if path_params: [e2e-llm-inference-service] path_params = self.sanitize_for_serialization(path_params) [e2e-llm-inference-service] path_params = self.parameters_to_tuples(path_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] for k, v in path_params: [e2e-llm-inference-service] # specified safe chars, encode everything [e2e-llm-inference-service] resource_path = resource_path.replace( [e2e-llm-inference-service] '{%s}' % k, [e2e-llm-inference-service] quote(str(v), safe=config.safe_chars_for_path_param) [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # query parameters [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] query_params = self.sanitize_for_serialization(query_params) [e2e-llm-inference-service] query_params = self.parameters_to_tuples(query_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] [e2e-llm-inference-service] # post parameters [e2e-llm-inference-service] if post_params or files: [e2e-llm-inference-service] post_params = post_params if post_params else [] [e2e-llm-inference-service] post_params = self.sanitize_for_serialization(post_params) [e2e-llm-inference-service] post_params = self.parameters_to_tuples(post_params, [e2e-llm-inference-service] collection_formats) [e2e-llm-inference-service] post_params.extend(self.files_parameters(files)) [e2e-llm-inference-service] [e2e-llm-inference-service] # auth setting [e2e-llm-inference-service] self.update_params_for_auth(header_params, query_params, auth_settings) [e2e-llm-inference-service] [e2e-llm-inference-service] # body [e2e-llm-inference-service] if body: [e2e-llm-inference-service] body = self.sanitize_for_serialization(body) [e2e-llm-inference-service] [e2e-llm-inference-service] # request url [e2e-llm-inference-service] if _host is None: [e2e-llm-inference-service] url = self.configuration.host + resource_path [e2e-llm-inference-service] else: [e2e-llm-inference-service] # use server/host defined in path or operation instead [e2e-llm-inference-service] url = _host + resource_path [e2e-llm-inference-service] [e2e-llm-inference-service] # perform request and return response [e2e-llm-inference-service] > response_data = self.request( [e2e-llm-inference-service] method, url, query_params=query_params, headers=header_params, [e2e-llm-inference-service] post_params=post_params, body=body, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:180: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = 'https://afb35e615998e4024955f70c61cd36d7-c1814657712f161b.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-with-lora-adapters-bbeb2681/llminferenceserviceconfigs' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] post_params = [] [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-no-scheduler-lora-multip-343dff1f', 'namespace': 'e2e-test-llm-with-lora-adapters-bbeb2681'}, 'spec': {'router': {'route': {}}}} [e2e-llm-inference-service] _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] post_params=None, body=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Makes the HTTP request using RESTClient.""" [e2e-llm-inference-service] if method == "GET": [e2e-llm-inference-service] return self.rest_client.GET(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif method == "HEAD": [e2e-llm-inference-service] return self.rest_client.HEAD(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif method == "OPTIONS": [e2e-llm-inference-service] return self.rest_client.OPTIONS(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout) [e2e-llm-inference-service] elif method == "POST": [e2e-llm-inference-service] > return self.rest_client.POST(url, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] post_params=post_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/api_client.py:391: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] url = 'https://afb35e615998e4024955f70c61cd36d7-c1814657712f161b.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-with-lora-adapters-bbeb2681/llminferenceserviceconfigs' [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] query_params = [], post_params = [] [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-no-scheduler-lora-multip-343dff1f', 'namespace': 'e2e-test-llm-with-lora-adapters-bbeb2681'}, 'spec': {'router': {'route': {}}}} [e2e-llm-inference-service] _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def POST(self, url, headers=None, query_params=None, post_params=None, [e2e-llm-inference-service] body=None, _preload_content=True, _request_timeout=None): [e2e-llm-inference-service] > return self.request("POST", url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] query_params=query_params, [e2e-llm-inference-service] post_params=post_params, [e2e-llm-inference-service] _preload_content=_preload_content, [e2e-llm-inference-service] _request_timeout=_request_timeout, [e2e-llm-inference-service] body=body) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:279: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = 'https://afb35e615998e4024955f70c61cd36d7-c1814657712f161b.elb.us-east-1.amazonaws.com:6443/apis/serving.kserve.io/v1alpha1/namespaces/e2e-test-llm-with-lora-adapters-bbeb2681/llminferenceserviceconfigs' [e2e-llm-inference-service] query_params = [] [e2e-llm-inference-service] headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/32.0.1/python'} [e2e-llm-inference-service] body = {'apiVersion': 'serving.kserve.io/v1alpha1', 'kind': 'LLMInferenceServiceConfig', 'metadata': {'name': 'router-no-scheduler-lora-multip-343dff1f', 'namespace': 'e2e-test-llm-with-lora-adapters-bbeb2681'}, 'spec': {'router': {'route': {}}}} [e2e-llm-inference-service] post_params = {}, _preload_content = True, _request_timeout = None [e2e-llm-inference-service] [e2e-llm-inference-service] def request(self, method, url, query_params=None, headers=None, [e2e-llm-inference-service] body=None, post_params=None, _preload_content=True, [e2e-llm-inference-service] _request_timeout=None): [e2e-llm-inference-service] """Perform requests. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: http request method [e2e-llm-inference-service] :param url: http request url [e2e-llm-inference-service] :param query_params: query parameters in the url [e2e-llm-inference-service] :param headers: http request headers [e2e-llm-inference-service] :param body: request json body, for `application/json` [e2e-llm-inference-service] :param post_params: request post parameters, [e2e-llm-inference-service] `application/x-www-form-urlencoded` [e2e-llm-inference-service] and `multipart/form-data` [e2e-llm-inference-service] :param _preload_content: if False, the urllib3.HTTPResponse object will [e2e-llm-inference-service] be returned without reading/decoding response [e2e-llm-inference-service] data. Default is True. [e2e-llm-inference-service] :param _request_timeout: timeout setting for this request. If one [e2e-llm-inference-service] number provided, it will be total request [e2e-llm-inference-service] timeout. It can also be a pair (tuple) of [e2e-llm-inference-service] (connection, read) timeouts. [e2e-llm-inference-service] """ [e2e-llm-inference-service] method = method.upper() [e2e-llm-inference-service] assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT', [e2e-llm-inference-service] 'PATCH', 'OPTIONS'] [e2e-llm-inference-service] [e2e-llm-inference-service] if post_params and body: [e2e-llm-inference-service] raise ApiValueError( [e2e-llm-inference-service] "body parameter cannot be used with post_params parameter." [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] post_params = post_params or {} [e2e-llm-inference-service] headers = headers or {} [e2e-llm-inference-service] [e2e-llm-inference-service] timeout = None [e2e-llm-inference-service] if _request_timeout: [e2e-llm-inference-service] if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)): # noqa: E501,F821 [e2e-llm-inference-service] timeout = urllib3.Timeout(total=_request_timeout) [e2e-llm-inference-service] elif (isinstance(_request_timeout, tuple) and [e2e-llm-inference-service] len(_request_timeout) == 2): [e2e-llm-inference-service] timeout = urllib3.Timeout( [e2e-llm-inference-service] connect=_request_timeout[0], read=_request_timeout[1]) [e2e-llm-inference-service] [e2e-llm-inference-service] if 'Content-Type' not in headers: [e2e-llm-inference-service] headers['Content-Type'] = 'application/json' [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE` [e2e-llm-inference-service] if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']: [e2e-llm-inference-service] if query_params: [e2e-llm-inference-service] url += '?' + urlencode(query_params) [e2e-llm-inference-service] if (re.search('json', headers['Content-Type'], re.IGNORECASE) or [e2e-llm-inference-service] headers['Content-Type'] == 'application/apply-patch+yaml'): [e2e-llm-inference-service] if headers['Content-Type'] == 'application/json-patch+json': [e2e-llm-inference-service] if not isinstance(body, list): [e2e-llm-inference-service] headers['Content-Type'] = \ [e2e-llm-inference-service] 'application/strategic-merge-patch+json' [e2e-llm-inference-service] request_body = None [e2e-llm-inference-service] if body is not None: [e2e-llm-inference-service] request_body = json.dumps(body) [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'application/x-www-form-urlencoded': # noqa: E501 [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=False, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] elif headers['Content-Type'] == 'multipart/form-data': [e2e-llm-inference-service] # must del headers['Content-Type'], or the correct [e2e-llm-inference-service] # Content-Type which generated by urllib3 will be [e2e-llm-inference-service] # overwritten. [e2e-llm-inference-service] del headers['Content-Type'] [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] fields=post_params, [e2e-llm-inference-service] encode_multipart=True, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] # Pass a `string` parameter directly in the body to support [e2e-llm-inference-service] # other content types than Json when `body` argument is [e2e-llm-inference-service] # provided in serialized form [e2e-llm-inference-service] elif isinstance(body, str) or isinstance(body, bytes): [e2e-llm-inference-service] request_body = body [e2e-llm-inference-service] r = self.pool_manager.request( [e2e-llm-inference-service] method, url, [e2e-llm-inference-service] body=request_body, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] else: [e2e-llm-inference-service] # Cannot generate the request from given parameters [e2e-llm-inference-service] msg = """Cannot prepare a request message for provided [e2e-llm-inference-service] arguments. Please check that your arguments match [e2e-llm-inference-service] declared content type.""" [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] # For `GET`, `HEAD` [e2e-llm-inference-service] else: [e2e-llm-inference-service] r = self.pool_manager.request(method, url, [e2e-llm-inference-service] fields=query_params, [e2e-llm-inference-service] preload_content=_preload_content, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] headers=headers) [e2e-llm-inference-service] except urllib3.exceptions.SSLError as e: [e2e-llm-inference-service] msg = "{0}\n{1}".format(type(e).__name__, str(e)) [e2e-llm-inference-service] raise ApiException(status=0, reason=msg) [e2e-llm-inference-service] [e2e-llm-inference-service] if _preload_content: [e2e-llm-inference-service] r = RESTResponse(r) [e2e-llm-inference-service] [e2e-llm-inference-service] # In the python 3, the response.data is bytes. [e2e-llm-inference-service] # we need to decode it to string. [e2e-llm-inference-service] if six.PY3: [e2e-llm-inference-service] r.data = r.data.decode('utf8') [e2e-llm-inference-service] [e2e-llm-inference-service] # log response body [e2e-llm-inference-service] logger.debug("response body: %s", r.data) [e2e-llm-inference-service] [e2e-llm-inference-service] if not 200 <= r.status <= 299: [e2e-llm-inference-service] > raise ApiException(http_resp=r) [e2e-llm-inference-service] E kubernetes.client.exceptions.ApiException: (500) [e2e-llm-inference-service] E Reason: Internal Server Error [e2e-llm-inference-service] E HTTP response headers: HTTPHeaderDict({'Audit-Id': 'e4acfe46-6278-4461-9014-5234ba67c818', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'bf546a5c-3f60-4261-9b3f-c3b9e58fb8c8', 'X-Kubernetes-Pf-Prioritylevel-Uid': '39b8073e-7e81-4403-a6ca-a29718cbfcdd', 'Date': 'Mon, 27 Jul 2026 18:12:43 GMT', 'Content-Length': '833'}) [e2e-llm-inference-service] E HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha1.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/validate-serving-kserve-io-v1alpha1-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha1.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/validate-serving-kserve-io-v1alpha1-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\""}]},"code":500} [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/kubernetes/client/rest.py:238: ApiException [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-with-lora-adapters-bbeb2681 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-with-lora-adapters-bbeb2681 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-with-lora-adapters-bbeb2681 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-with-lora-adapters-bbeb2681 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-with-lora-adapters-bbeb2681 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-with-lora-adapters-bbeb2681 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_with_lora_adapters] [2026-07-27T18:12:43.456137] start - args=(), kwargs={'test_case': LoRATestCase(base_refs=['router-no-scheduler', 'workload-single-cpu', 'model-fb-opt-125m-with-multiple-lora'], prompt='Explain machine learning in simple terms.', expected_adapter_names=['lora-adapter-1', 'lora-adapter-2'], service_name='lora-multiple-adapters-test', endpoint='/v1/completions', max_tokens=100, wait_timeout=900, response_timeout=60), 'test_namespace': 'e2e-test-llm-with-lora-adapters-bbeb2681'} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_lora_adapters:test_llm_lora_adapters.py:118 Creating LLMInferenceServiceConfig: router-no-scheduler-lora-multip-343dff1f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-no-scheduler-lora-multip-343dff1f in namespace e2e-test-llm-with-lora-adapters-bbeb2681 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-no-scheduler-lora-multip-343dff1f [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:243 # Diagnostics for 'lora-multiple-adapters-test' in 'e2e-test-llm-with-lora-adapters-bbeb2681' [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:244 --- [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:247 # LLMInferenceService lora-multiple-adapters-test [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:258 # failed to dump LLMInferenceService: (404) [e2e-llm-inference-service] Reason: Not Found [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '601e2f2b-d98a-4800-ba14-d5795bb845a5', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'bf546a5c-3f60-4261-9b3f-c3b9e58fb8c8', 'X-Kubernetes-Pf-Prioritylevel-Uid': '39b8073e-7e81-4403-a6ca-a29718cbfcdd', 'Date': 'Mon, 27 Jul 2026 18:12:43 GMT', 'Content-Length': '300'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"llminferenceservices.serving.kserve.io \"lora-multiple-adapters-test\" not found","reason":"NotFound","details":{"name":"lora-multiple-adapters-test","group":"serving.kserve.io","kind":"llminferenceservices"},"code":404} [e2e-llm-inference-service] [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:64 ℹ️ # No events found in namespace e2e-test-llm-with-lora-adapters-bbeb2681 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:166 # no pods found in e2e-test-llm-with-lora-adapters-bbeb2681 matching app.kubernetes.io/part-of=llminferenceservice,app.kubernetes.io/name=lora-multiple-adapters-test [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_with_lora_adapters] [2026-07-27T18:12:44.840565] end - ❌ 1.384s: (500) [e2e-llm-inference-service] Reason: Internal Server Error [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': 'e4acfe46-6278-4461-9014-5234ba67c818', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': 'bf546a5c-3f60-4261-9b3f-c3b9e58fb8c8', 'X-Kubernetes-Pf-Prioritylevel-Uid': '39b8073e-7e81-4403-a6ca-a29718cbfcdd', 'Date': 'Mon, 27 Jul 2026 18:12:43 GMT', 'Content-Length': '833'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha1.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/validate-serving-kserve-io-v1alpha1-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\"","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook \"llminferenceserviceconfig.kserve-webhook-server.v1alpha1.validator\": failed to call webhook: Post \"https://llmisvc-webhook-server-service.kserve.svc:443/validate-serving-kserve-io-v1alpha1-llminferenceserviceconfig?timeout=10s\": no endpoints available for service \"llmisvc-webhook-server-service\""}]},"code":500} [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-with-lora-adapters-bbeb2681 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_hpa_lws[router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-hpa'], prompt='KSer... {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_hpa [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-lws", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-hpa", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-hpa-lws", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_hpa_lws(test_case: TestCase): [e2e-llm-inference-service] """HPA + LWS: HPA exists with WVA annotations; pods scale under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:670: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-hpa'], prompt='KSer... {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...ale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-27T17:56:47.629162', start_time = 1785175007.6294417 [e2e-llm-inference-service] duration = 900.3491790294647, timestamp_end = '2026-07-27T18:11:47.978670' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f4e06fc99e0> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:57:44Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:57:44Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:57:22Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:57:55Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:57:55Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:57:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:57:55Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:57:44Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:57:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-hpa-lw-1aa98714 in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-hpa-lw-1aa98714 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-hpa-lw-1aa98714 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-fe7a55cc in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-fe7a55cc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-fe7a55cc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-b29acdba in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-b29acdba [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-b29acdba [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-lws-b344a3ff in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-lws-b344a3ff [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-lws-b344a3ff [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_hpa_lws] [2026-07-27T17:56:47.268784] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-hpa'], prompt='KServe is a', service_name='autoscale-hpa-lws', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-27T17:56:47.281593] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-27T17:56:47.629013] end - ✅ in 0.347s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-27T17:56:47.629162] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:57:22Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:57:22Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:57:22Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:57:22Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:57:22Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:57:22Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:57:22Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:57:22Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:57:22Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:57:44Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:57:44Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:57:22Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:57:44Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:57:44Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:57:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:57:44Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:57:44Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:57:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:57:44Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:57:44Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:57:22Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:57:55Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:57:55Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:57:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:57:55Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:57:44Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:57:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:57:44Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:57:44Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:57:22Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:57:55Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:57:55Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:57:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:57:55Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:57:44Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:57:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-27T18:11:47.978670] end - ❌ 900.349s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:57:44Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:57:44Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:57:22Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:57:55Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:57:55Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:57:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:57:55Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:57:44Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:57:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-27T18:11:47.979271] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 3 pod(s) for autoscale-hpa-lws still terminating: ['autoscale-hpa-lws-kserve-mn-0', 'autoscale-hpa-lws-kserve-mn-0-1', 'autoscale-hpa-lws-kserve-router-scheduler-584b9979f5-dmjq2'] [e2e-llm-inference-service] assert not ['autoscale-hpa-lws-kserve-mn-0', 'autoscale-hpa-lws-kserve-mn-0-1', 'autoscale-hpa-lws-kserve-router-scheduler-584b9979f5-dmjq2'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-hpa-lws still terminating: ['autoscale-hpa-lws-kserve-router-scheduler-584b9979f5-dmjq2'] [e2e-llm-inference-service] assert not ['autoscale-hpa-lws-kserve-router-scheduler-584b9979f5-dmjq2'] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: 1 pod(s) for autoscale-hpa-lws still terminating: ['autoscale-hpa-lws-kserve-router-scheduler-584b9979f5-dmjq2'] [e2e-llm-inference-service] assert not ['autoscale-hpa-lws-kserve-router-scheduler-584b9979f5-dmjq2'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-27T18:13:48.686178] end - ✅ in 120.705s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_hpa_lws] [2026-07-27T18:13:48.686242] end - ❌ 1021.417s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T17:57:44Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T17:57:44Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T17:57:22Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T17:57:55Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T17:57:55Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T17:57:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T17:57:55Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:57:44Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T17:57:44Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_keda_lws[router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-keda'], prompt='KSe... {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_keda [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-lws", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-keda", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-keda-lws", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_keda_lws(test_case: TestCase): [e2e-llm-inference-service] """KEDA + LWS: ScaledObject exists with WVA annotations; pods scale under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:728: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-keda'], prompt='KSe... {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-27T18:13:50.228840', start_time = 1785176030.2291372 [e2e-llm-inference-service] duration = 900.2005784511566, timestamp_end = '2026-07-27T18:28:50.429718' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f4e06fcaf20> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:14:37Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:14:37Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:14:08Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:14:57Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:14:57Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:14:37Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:14:57Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:14:37Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:14:37Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-keda-lws-e541a132 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-keda-l-78828c4a in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-keda-l-78828c4a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-keda-l-78828c4a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-1696d0b7 in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-1696d0b7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-1696d0b7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-231d315d in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-231d315d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-231d315d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-keda-autoscale-keda-lws-1337f511 in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-keda-autoscale-keda-lws-1337f511 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-keda-autoscale-keda-lws-1337f511 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_keda_lws] [2026-07-27T18:13:49.868478] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-keda'], prompt='KServe is a', service_name='autoscale-keda-lws', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-keda-lws-e541a132', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-27T18:13:49.881078] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-27T18:13:50.228693] end - ✅ in 0.347s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-27T18:13:50.228840] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:14:08Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:14:08Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:14:08Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:14:08Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:14:08Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:14:08Z', 'message': 'ScaledObject not yet visible in cache', 'reason': 'ScaledObjectProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:14:08Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:14:08Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:14:08Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:14:37Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:14:37Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:14:08Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:14:37Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:14:37Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:14:37Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:14:37Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:14:37Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:14:37Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:14:37Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:14:37Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:14:08Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:14:57Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:14:57Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:14:37Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:14:57Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:14:37Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:14:37Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:14:37Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:14:37Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:14:08Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:14:57Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:14:57Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:14:37Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:14:57Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:14:37Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:14:37Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-27T18:28:50.429718] end - ❌ 900.201s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:14:37Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:14:37Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:14:08Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:14:57Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:14:57Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:14:37Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:14:57Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:14:37Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:14:37Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-27T18:28:50.429798] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 3 pod(s) for autoscale-keda-lws still terminating: ['autoscale-keda-lws-kserve-mn-0', 'autoscale-keda-lws-kserve-mn-0-1', 'autoscale-keda-lws-kserve-router-scheduler-6cd9466d4f-sbzf8'] [e2e-llm-inference-service] assert not ['autoscale-keda-lws-kserve-mn-0', 'autoscale-keda-lws-kserve-mn-0-1', 'autoscale-keda-lws-kserve-router-scheduler-6cd9466d4f-sbzf8'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-keda-lws still terminating: ['autoscale-keda-lws-kserve-router-scheduler-6cd9466d4f-sbzf8'] [e2e-llm-inference-service] assert not ['autoscale-keda-lws-kserve-router-scheduler-6cd9466d4f-sbzf8'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-27T18:29:20.767057] end - ✅ in 30.337s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_keda_lws] [2026-07-27T18:29:20.767203] end - ❌ 930.898s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-27T18:14:37Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-27T18:14:37Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-27T18:14:08Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-27T18:14:57Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-27T18:14:57Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-27T18:14:37Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-27T18:14:57Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:14:37Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-27T18:14:37Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-keda-lws-e541a132 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] =============================== warnings summary =============================== [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] /workspace/source/python/kserve/.venv/lib64/python3.11/site-packages/pytest_asyncio/plugin.py:761: DeprecationWarning: The event_loop fixture provided by pytest-asyncio has been redefined in [e2e-llm-inference-service] /workspace/source/test/e2e/conftest.py:43 [e2e-llm-inference-service] Replacing the event_loop fixture with a custom implementation is deprecated [e2e-llm-inference-service] and will lead to errors in the future. [e2e-llm-inference-service] If you want to request an asyncio event loop with a scope other than function [e2e-llm-inference-service] scope, use the "scope" argument to the asyncio mark when marking the tests. [e2e-llm-inference-service] If you want to return different types of event loops, use the event_loop_policy [e2e-llm-inference-service] fixture. [e2e-llm-inference-service] [e2e-llm-inference-service] warnings.warn( [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py:47: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py:47: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py:40: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html [e2e-llm-inference-service] - generated xml file: /workspace/artifacts-dir/junit_e2e-llminferenceservice_and_cluster_cpu_and_not_autoscaling_and_not_tracing.xml - [e2e-llm-inference-service] --------------------------------- JSON report ---------------------------------- [e2e-llm-inference-service] report saved to: /workspace/artifacts-dir/e2e_results-llminferenceservice_and_cluster_cpu_and_not_autoscaling_and_not_tracing.json [e2e-llm-inference-service] =========================== short test summary info ============================ [e2e-llm-inference-service] FAILED llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] FAILED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] FAILED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 10 failures !!!!!!!!!!!!!!!!!!!!!!!!!! [e2e-llm-inference-service] !!!!!!!!!!!! xdist.dsession.Interrupted: stopping after 5 failures !!!!!!!!!!!!! [e2e-llm-inference-service] ====== 10 failed, 34 passed, 3 skipped, 24 warnings in 4545.54s (1:15:45) ====== [must-gather] [must-gather ] OUT 2026-07-27T18:29:22.691434674Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 67b080d3-17dc-4b35-8a6a-b965aa8fce53 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-27T18:29:22.742674326Z namespace/openshift-must-gather-pftgl created [must-gather] [must-gather ] OUT 2026-07-27T18:29:22.753517834Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-6gbbc created [must-gather] [must-gather ] OUT 2026-07-27T18:29:22.77782145Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:30.604194280Z [disk usage checker] Started [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:30.607526728Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:30.865397714Z Error from server (NotFound): namespaces "redhat-ods-operator" not found [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:31.031903096Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:31.031935419Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:31.031935419Z namespaces "redhat-ods-operator" not found [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:31.034267373Z Error getting logs from redhat-ods-operator [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:31.209186142Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:31.209213759Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:31.209213759Z namespaces "redhat-ods-monitoring" not found [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:31.211704765Z Error getting logs from redhat-ods-monitoring [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:31.378773730Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:31.378801749Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:31.378801749Z namespaces "redhat-ods-applications" not found [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:31.381040040Z Error getting logs from redhat-ods-applications [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:31.554911343Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:31.554942172Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:31.554942172Z namespaces "rhods-notebooks" not found [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:31.557307828Z Error getting logs from rhods-notebooks [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:31.718558261Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:31.718598447Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:31.718598447Z namespaces "rhoai-model-registries" not found [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:31.721555061Z Error getting logs from rhoai-model-registries [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:31.886490759Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:31.886521268Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:31.886521268Z namespaces "istio-system" not found [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:31.889009055Z Error getting logs from istio-system [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:32.052976850Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:32.053008367Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:32.053008367Z namespaces "knative-serving" not found [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:32.055472613Z Error getting logs from knative-serving [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:32.221021887Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:32.221052632Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:32.221052632Z namespaces "redhat-ods-applications-auth-provider" not found [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:32.223222506Z Error getting logs from redhat-ods-applications-auth-provider [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:32.513450195Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:32.801142733Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:32.942047917Z error: the server doesn't have a resource type "auths" [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:33.024372417Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:33.026514825Z Error collecting info from [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:33.158342550Z error: the server doesn't have a resource type "monitorings" [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:33.239133641Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:33.241241858Z Error collecting info from [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:33.372357129Z error: the server doesn't have a resource type "featuretrackers" [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:33.451562810Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:33.453676815Z Error collecting info from [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:33.582601005Z error: the server doesn't have a resource type "codeflares" [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:33.662298146Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:33.664414577Z Error collecting info from [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:33.798341914Z error: the server doesn't have a resource type "dashboards" [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:33.881425194Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:33.883501770Z Error collecting info from [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:34.018547054Z error: the server doesn't have a resource type "datasciencepipelines" [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:34.098246815Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:34.100306867Z Error collecting info from [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:34.229639961Z error: the server doesn't have a resource type "feastoperators" [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:34.318497203Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:34.320768955Z Error collecting info from [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:34.455138871Z error: the server doesn't have a resource type "kserves" [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:34.535116769Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:34.537173051Z Error collecting info from [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:34.672258663Z error: the server doesn't have a resource type "kueues" [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:34.753440127Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:34.755547266Z Error collecting info from [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:34.893566044Z error: the server doesn't have a resource type "modelcontrollers" [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:34.972466926Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:34.974439847Z Error collecting info from [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:35.109311768Z error: the server doesn't have a resource type "modelmeshservings" [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:35.188396040Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:35.190479987Z Error collecting info from [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:35.320650568Z error: the server doesn't have a resource type "modelregistries" [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:35.403016662Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:35.405368393Z Error collecting info from [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:35.537329891Z error: the server doesn't have a resource type "rays" [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:35.611986087Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:35.616802364Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:35.619000850Z Error collecting info from [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:35.751137885Z error: the server doesn't have a resource type "trainingoperators" [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:35.836224664Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:35.839912959Z Error collecting info from [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:35.971588884Z error: the server doesn't have a resource type "trustyais" [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:36.049931808Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:36.052037222Z Error collecting info from [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:36.185410164Z error: the server doesn't have a resource type "workbenches" [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:36.269861709Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:36.272208585Z Error collecting info from [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:36.405648894Z error: the server doesn't have a resource type "hardwareprofiles" [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:36.486393704Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:36.488556468Z Error collecting info from [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:36.619743063Z error: the server doesn't have a resource type "llamastackoperators" [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:36.701549651Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:36.703818933Z Error collecting info from [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:37.401402181Z error: the server doesn't have a resource type "predictors" [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:37.528594579Z error: the server doesn't have a resource type "localmodelnodegroups" [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:37.656965127Z error: the server doesn't have a resource type "smcp" [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:37.790665837Z error: the server doesn't have a resource type "smm" [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:37.923931245Z error: the server doesn't have a resource type "smmr" [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:38.514375650Z error: the server doesn't have a resource type "knativeservings" [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:38.645377856Z error: the server doesn't have a resource type "configurations" [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:38.780333217Z error: the server doesn't have a resource type "routes" [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:38.916682529Z error: the server doesn't have a resource type "services" [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:39.044998366Z error: the server doesn't have a resource type "revisions" [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:39.612214665Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:39.783797228Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:39.873331263Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:39.949162002Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:40.108939596Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:40.251545712Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:40.351272613Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:40.430149695Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:40.591316831Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:40.619199413Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:40.738721742Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:40.828915698Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:40.904709977Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:41.067972988Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:41.305983925Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:41.388203592Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:41.462482834Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:41.619803605Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:41.765741150Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:41.857571744Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:41.935478566Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:42.096176637Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:42.236387844Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:42.320294172Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:42.392983854Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:42.556405649Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:42.706163679Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:42.792895693Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:42.870078079Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:43.033464285Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:43.213918841Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:43.294780806Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:43.366657464Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:43.530144559Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:43.669224263Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:43.749741798Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:43.826581188Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:43.986233648Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:44.137168323Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:44.222750407Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:44.297114156Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:44.463525074Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:44.608682757Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:44.754532078Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:44.829515381Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:44.996700079Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:45.141081813Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:45.228814542Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:45.308363305Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:45.472752603Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:45.616331200Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:45.624088813Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:45.697094817Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:45.769666579Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:45.936942011Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:46.108223831Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:46.191002190Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:46.269332961Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:46.433121100Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:47.178159945Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:47.350137372Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:47.462585427Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:47.641548462Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:48.415243374Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:48.589209223Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:48.702579018Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:48.877050164Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:49.603619778Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:49.780103780Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:49.895259843Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:50.063313631Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:50.628546174Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:50.820920511Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:50.996933998Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:51.107326001Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:51.275162429Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:52.084852102Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:52.271251969Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:52.412214113Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:52.592083258Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:53.430504225Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:53.627885764Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:53.753671681Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:53.919646722Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:54.690185604Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:54.864577740Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:54.982782069Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:55.144489956Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:55.633199628Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:56.003006113Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:56.194458666Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:56.314193645Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:56.482274225Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:57.243999631Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:57.427679353Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:57.543288220Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:57.704625664Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:58.481268552Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:58.661517133Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:58.778751356Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:58.943987500Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:59.721015902Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:29:59.904148151Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:00.026346304Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:00.195043009Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:00.637822334Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:00.981076489Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:01.161892461Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:01.314501045Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:01.479882688Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:02.352199784Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:02.608085700Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:02.729466476Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:02.901359847Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:03.718146418Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:03.904073420Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:04.041730930Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:04.203431915Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:05.000239821Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:05.178403073Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:05.299850781Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:05.489060891Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:05.642641356Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:06.283963522Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:06.473038041Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:06.594576400Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:06.761418870Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:07.555649077Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:07.744645631Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:07.866303506Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:08.029179532Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:08.808693893Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:08.990273976Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:09.112644014Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:09.276818423Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:10.019467416Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:10.197006748Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:10.316032053Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:10.477477027Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:10.647529148Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:11.242210945Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:11.429560213Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:11.547956392Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:11.710747274Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:12.482040700Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:12.669862964Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:12.786114530Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:12.945824990Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:13.767332215Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:13.950693749Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:14.068425560Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:14.230439770Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:15.013086208Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:15.192604882Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:15.321716171Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:15.484462320Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:15.652130149Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:16.257280636Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:16.445387545Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:16.565807456Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:16.732558232Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:17.464497693Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:17.656008023Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:17.779410181Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:17.941817596Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:18.756715999Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:18.939313509Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:19.058472549Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:19.225897706Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:20.002938382Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:20.187812844Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:20.311018579Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:20.478596712Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:20.656791767Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:21.264574100Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:21.450781880Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:21.572846207Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:21.742066730Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:22.566609509Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:22.757788212Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:22.884198570Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:23.051101329Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:23.822983689Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:24.006007357Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:24.126048154Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:24.292135440Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:25.083446631Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:25.265077634Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:25.390964695Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:25.564510286Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:25.661541749Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:26.345879526Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:26.532163347Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:26.660911047Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:26.830198969Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:27.609187285Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:27.795272965Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:27.924370275Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:28.087542677Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:28.876987408Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:29.072959258Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:29.197893321Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:29.376487126Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:30.177797190Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:30.368729133Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:30.493207893Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:30.661417545Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:30.666042820Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:31.471955832Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:31.663807297Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:31.791771776Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:31.960365488Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:32.754110083Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:32.956065512Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:33.083815335Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:33.259234305Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:34.040974080Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:34.232005118Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:34.365462993Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:34.535490822Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:35.320382066Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:35.513593317Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:35.636189104Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:35.672005093Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:35.813094949Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:36.616317803Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:36.803034160Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:36.944036068Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:37.115131473Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:37.912969908Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:38.099894598Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:38.223483129Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:38.395700938Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:39.178358239Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:39.361867959Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:39.484492160Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:39.647417604Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:40.429729076Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:40.616640128Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:40.676450465Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:40.743053724Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:40.912359427Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:41.696420763Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:41.887613619Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:42.018294152Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:42.187350868Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:42.979683037Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:43.172579348Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:43.295824600Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:43.462702505Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:44.245577410Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:44.434752785Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:44.564332386Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:44.736044093Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:45.539193865Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:45.680970785Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:45.722933466Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:45.853994816Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:46.024395455Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:46.795534789Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:46.996703622Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:47.121775204Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:47.303096083Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:48.095359546Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:48.279731541Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:48.400101737Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:48.567889922Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:49.351102456Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:49.546561740Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:49.671340177Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:49.843339419Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:50.611664558Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:50.687868396Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:50.795888996Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:50.919562252Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:51.085940699Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:51.869721650Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:52.061273937Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:52.192725847Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:52.362611574Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:53.154486256Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:53.348205955Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:53.468548741Z Wrote inspect data to must-gather. [must-gather] [must-gather-bgn56] POD 2026-07-27T18:30:53.490303085Z Caches written to disk [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.498753936Z waiting for gather to complete [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.500812173Z downloading gather output [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.764609073Z receiving incremental file list [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.777534471Z ./ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.777629854Z aggregated-discovery-api.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.777929681Z aggregated-discovery-apis.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.779053577Z event-filter.html [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.779417815Z timestamp [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.779574749Z version [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.784248056Z cluster-scoped-resources/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.784261077Z cluster-scoped-resources/datasciencecluster.opendatahub.io/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.784266487Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.784324018Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/test-dsc.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.78440441Z cluster-scoped-resources/dscinitialization.opendatahub.io/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.78441779Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.784460441Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/test-dsci.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.784540953Z namespaces/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.784550403Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.784592024Z namespaces/kserve-ci-e2e-test/kserve-ci-e2e-test.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.784658296Z namespaces/kserve-ci-e2e-test/apps.openshift.io/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.784687207Z namespaces/kserve-ci-e2e-test/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.784736098Z namespaces/kserve-ci-e2e-test/apps/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.784777559Z namespaces/kserve-ci-e2e-test/apps/daemonsets.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.784867951Z namespaces/kserve-ci-e2e-test/apps/deployments.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.784976903Z namespaces/kserve-ci-e2e-test/apps/replicasets.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.785061725Z namespaces/kserve-ci-e2e-test/apps/statefulsets.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.785109776Z namespaces/kserve-ci-e2e-test/autoscaling/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.785153137Z namespaces/kserve-ci-e2e-test/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.785188848Z namespaces/kserve-ci-e2e-test/batch/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.785225309Z namespaces/kserve-ci-e2e-test/batch/cronjobs.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.785309351Z namespaces/kserve-ci-e2e-test/batch/jobs.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.785360092Z namespaces/kserve-ci-e2e-test/build.openshift.io/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.785401093Z namespaces/kserve-ci-e2e-test/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.785482235Z namespaces/kserve-ci-e2e-test/build.openshift.io/builds.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.785533606Z namespaces/kserve-ci-e2e-test/core/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.785567557Z namespaces/kserve-ci-e2e-test/core/configmaps.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.78569212Z namespaces/kserve-ci-e2e-test/core/endpoints.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.785775382Z namespaces/kserve-ci-e2e-test/core/events.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.785852034Z namespaces/kserve-ci-e2e-test/core/persistentvolumeclaims.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.785948876Z namespaces/kserve-ci-e2e-test/core/pods.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.786026347Z namespaces/kserve-ci-e2e-test/core/replicationcontrollers.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.78614875Z namespaces/kserve-ci-e2e-test/core/secrets.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.786327874Z namespaces/kserve-ci-e2e-test/core/services.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.786375886Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.786411806Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.786461017Z namespaces/kserve-ci-e2e-test/image.openshift.io/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.786503169Z namespaces/kserve-ci-e2e-test/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.78654553Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.786602611Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.786687243Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.786740564Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.786755244Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.786821566Z namespaces/kserve-ci-e2e-test/networking.k8s.io/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.786850657Z namespaces/kserve-ci-e2e-test/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.786925088Z namespaces/kserve-ci-e2e-test/policy/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.786942839Z namespaces/kserve-ci-e2e-test/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.78700419Z namespaces/kserve-ci-e2e-test/route.openshift.io/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.787033751Z namespaces/kserve-ci-e2e-test/route.openshift.io/routes.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.787087502Z namespaces/kuadrant-system/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.787129043Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.787188984Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.787231475Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.787262706Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.787306877Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.787390689Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.787724097Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.788060544Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.788091035Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.788141016Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.788182667Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.788213148Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.78830347Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.788333211Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.788383692Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.788467334Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.788509985Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.788539005Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.78874204Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.788871323Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.789378275Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.789455557Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.789822625Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.789959948Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.790185073Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.790311886Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.790329147Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.79046758Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.790497091Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.790550912Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.790581632Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.790682935Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.790721646Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.790764237Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.790820308Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.790839849Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.790931481Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.790944491Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-s6nzs/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.790957431Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-s6nzs/authorino-7cf4765b85-s6nzs.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.791051953Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-s6nzs/authorino/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.791061024Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-s6nzs/authorino/authorino/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.791067104Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-s6nzs/authorino/authorino/logs/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.791096784Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-s6nzs/authorino/authorino/logs/current.log [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.802264632Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-s6nzs/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.802333454Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-s6nzs/authorino/authorino/logs/previous.log [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.802372835Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-n5pb8/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.802420186Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-n5pb8/authorino-operator-56fb6f6856-n5pb8.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.802520778Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-n5pb8/manager/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.802531699Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-n5pb8/manager/manager/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.802542589Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-n5pb8/manager/manager/logs/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.802574029Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-n5pb8/manager/manager/logs/current.log [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.802706003Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-n5pb8/manager/manager/logs/previous.log [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.802891717Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq5xn/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.802913567Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq5xn/dns-operator-controller-manager-ccbdf686-qq5xn.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.803005809Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq5xn/manager/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.8030168Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq5xn/manager/manager/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.80302716Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq5xn/manager/manager/logs/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.803065141Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq5xn/manager/manager/logs/current.log [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.803174853Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq5xn/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.803234105Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq5xn/manager/manager/logs/previous.log [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.803254965Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-c5j4w/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.803309966Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-c5j4w/kuadrant-console-plugin-598f86468b-c5j4w.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.803390318Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-c5j4w/kuadrant-console-plugin/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.803397719Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-c5j4w/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.803402108Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-c5j4w/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.803432789Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-c5j4w/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.803496041Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-c5j4w/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.803570792Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-c5j4w/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.803605513Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-fl2ps/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.803643314Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-fl2ps/kuadrant-operator-controller-manager-66866f5fcc-fl2ps.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.803737066Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-fl2ps/manager/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.803749367Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-fl2ps/manager/manager/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.803753607Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-fl2ps/manager/manager/logs/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.803797378Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-fl2ps/manager/manager/logs/current.log [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.81864626Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-fl2ps/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.818721492Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-fl2ps/manager/manager/logs/previous.log [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.818765793Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-cdzw4/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.818799504Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-cdzw4/limitador-limitador-85d9db4d5d-cdzw4.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.818905097Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-cdzw4/limitador/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.818918337Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-cdzw4/limitador/limitador/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.818927977Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-cdzw4/limitador/limitador/logs/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.818973748Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-cdzw4/limitador/limitador/logs/current.log [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.81905985Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-cdzw4/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.819133472Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-cdzw4/limitador/limitador/logs/previous.log [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.819161612Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-6fl6q/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.819208633Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-6fl6q/limitador-operator-controller-manager-774d94476f-6fl6q.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.819303766Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-6fl6q/manager/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.819312506Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-6fl6q/manager/manager/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.819317426Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-6fl6q/manager/manager/logs/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.819339936Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-6fl6q/manager/manager/logs/current.log [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.81949788Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-6fl6q/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.819567652Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-6fl6q/manager/manager/logs/previous.log [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.819610033Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.819639683Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.819690064Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.819743486Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.819790137Z namespaces/openshift-ingress/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.819872119Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.81992352Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.819979421Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.820020522Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.820067573Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.820150395Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.820403911Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.820615016Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.820656997Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.820712178Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.820763279Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.82080338Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.820909563Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.820944424Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.820995055Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.821081467Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.821136618Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.821155288Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.821606649Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.821709971Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.821945877Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.822019058Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.822270654Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.822404917Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.822609992Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.822714114Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.822756795Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.822835137Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.822880988Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.8229452Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.822986651Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.823068982Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.823113543Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.823157024Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.823212306Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.823252137Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.823299788Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.823306808Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-hkbqw/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.823348709Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-hkbqw/istiod-openshift-gateway-5677ccb49b-hkbqw.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.823465592Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-hkbqw/discovery/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.823472722Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-hkbqw/discovery/discovery/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.823476642Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-hkbqw/discovery/discovery/logs/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.823507503Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-hkbqw/discovery/discovery/logs/current.log [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.839357169Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-hkbqw/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.83944362Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-hkbqw/discovery/discovery/logs/previous.log [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.839496482Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.839568454Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.839727857Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c/istio-proxy/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.839736557Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c/istio-proxy/istio-proxy/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.839741718Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.839788559Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.841213241Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.841343064Z namespaces/openshift-ingress/pods/router-default-57898f8749-pb7sc/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.841400036Z namespaces/openshift-ingress/pods/router-default-57898f8749-pb7sc/router-default-57898f8749-pb7sc.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.841531889Z namespaces/openshift-ingress/pods/router-default-57898f8749-pb7sc/router/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.841540859Z namespaces/openshift-ingress/pods/router-default-57898f8749-pb7sc/router/router/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.841546269Z namespaces/openshift-ingress/pods/router-default-57898f8749-pb7sc/router/router/logs/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.84159038Z namespaces/openshift-ingress/pods/router-default-57898f8749-pb7sc/router/router/logs/current.log [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.841724603Z namespaces/openshift-ingress/pods/router-default-57898f8749-pb7sc/router/router/logs/previous.insecure.log [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.841813085Z namespaces/openshift-ingress/pods/router-default-57898f8749-pb7sc/router/router/logs/previous.log [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.841875387Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.841937178Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.84201034Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.842066821Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.848546711Z [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.848563201Z sent 2,829 bytes received 987,080 bytes 1,979,818.00 bytes/sec [must-gather] [must-gather-bgn56] OUT 2026-07-27T18:30:54.848569891Z total size is 14,464,738 speedup is 14.61 [must-gather] [must-gather ] OUT 2026-07-27T18:30:54.902145588Z namespace/openshift-must-gather-pftgl deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 67b080d3-17dc-4b35-8a6a-b965aa8fce53 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-27T18:30:55.015875653Z Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35f66492587d98a3cca9bf7ba4541c867770dc76b78b4cf9352e6715dca04502 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 67b080d3-17dc-4b35-8a6a-b965aa8fce53 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-27T18:30:55.034307989Z namespace/openshift-must-gather-q5775 created [must-gather] [must-gather ] OUT 2026-07-27T18:30:55.041972636Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-sqsrm created [must-gather] [must-gather ] OUT 2026-07-27T18:30:55.071298493Z pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35f66492587d98a3cca9bf7ba4541c867770dc76b78b4cf9352e6715dca04502 created [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:56.748630755Z [disk usage checker] Started [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:56.752123918Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:56.975895291Z Gathering data for ns/openshift-cluster-version... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:57.137948888Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:57.320886917Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:57.428350869Z Gathering data for ns/default... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:57.782133950Z Gathering data for ns/openshift... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:58.249021383Z Gathering data for ns/kube-system... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:59.107879301Z Gathering data for ns/openshift-etcd... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:59.485817157Z Wrote inspect data to must-gather. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:59.485868337Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:59.485868337Z namespaces "assisted-installer" not found [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:59.786018312Z Waiting on subprocesses to finish execution. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:59.793551267Z INFO: Gathering machine config daemon's old logs from all nodes [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:59.796107229Z Executing Istio gather script [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:59.797313369Z INFO: Collecting host service logs for crio [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:59.797627603Z INFO: Collecting host service logs for kubelet [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:59.797962458Z INFO: Collecting host service logs for rpm-ostreed [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:59.798043648Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:59.798260740Z INFO: Collecting host service logs for ostree-finalize-staged [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:59.798579655Z INFO: Collecting host service logs for machine-config-daemon-firstboot [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:59.798878383Z INFO: Collecting host service logs for machine-config-daemon-host [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:59.799181707Z INFO: Collecting host service logs for NetworkManager [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:59.799475729Z INFO: Collecting host service logs for openvswitch [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:59.799769931Z INFO: Collecting host service logs for ovs-configuration [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:59.800089345Z INFO: Collecting host service logs for ovsdb-server [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:59.800390144Z INFO: Collecting host service logs for ovs-vswitchd [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:59.800736762Z INFO: Waiting for worker host service log collection to complete ... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:59.810280455Z INFO: Gathering HAProxy config files [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:59.830208014Z INFO: Gathering on-disk MachineConfig from degraded nodes [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:30:59.951537818Z INFO: Waiting for node performance related collection to complete ... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:00.408798181Z INFO: namespace openshift-frr-k8s not detected. Skipping. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:00.817154783Z INFO: Waiting for on-disk MachineConfig collection to complete ... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:00.817250815Z INFO: on-disk MachineConfig config collection complete. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:01.057521548Z No resources found in openshift-etcd namespace. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:01.057752513Z Wrote inspect data to must-gather. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:01.220995843Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-2vf8b [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:01.220995843Z insights-runtime-extractor-gjcfv [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:01.220995843Z insights-runtime-extractor-l6rw6 [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:01.220995843Z insights-runtime-extractor-mz5cv [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:01.225284125Z INFO: "metallb-operator" not detected. Skipping. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:01.263279431Z INFO: "sriov-network-operator" not detected. Skipping. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:01.407112946Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:01.810604088Z No resources found [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:01.974776507Z INFO: Found 1 replicas - prometheus-k8s-0 [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:02.017977116Z INFO: Worker host service log collection to complete. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:02.040300416Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:02.358277082Z ERROR: No running kube-apiserver pods found [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:02.361513959Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:02.361513959Z See 'oc rsync -h' for help and examples [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:02.510615142Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:02.565511592Z Wrote inspect data to must-gather. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:02.684113765Z Wrote inspect data to must-gather. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:02.720742765Z Wrote inspect data to must-gather. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:02.764585881Z INFO: Waiting for HAProxy config collection to complete ... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:02.894115063Z INFO: Getting alertmanagers from prometheus-k8s-0 [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:02.967383980Z error: the server doesn't have a resource type "performanceprofile" [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:02.978637385Z error: the server doesn't have a resource type "multi-networkpolicy" [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:03.013884129Z Wrote inspect data to must-gather. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:03.042650897Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:03.126742055Z INFO: 'previous-logs' folder not found on ip-10-0-132-194.ec2.internal, skipping... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:03.127859691Z No resources found [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:03.387931398Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:03.424144675Z tar: Removing leading `/' from member names [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:03.442230171Z INFO: HAProxy config collection complete. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:03.586177012Z Wrote inspect data to must-gather. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:03.605004707Z Wrote inspect data to must-gather. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:03.628948027Z INFO: 'previous-logs' folder not found on ip-10-0-137-134.ec2.internal, skipping... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:03.802008032Z INFO: Getting rules from prometheus-k8s-0 [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:03.845461594Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:03.849876358Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:03.947581949Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:03.957891980Z error: the server doesn't have a resource type "machineconfigs" [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:04.173781307Z INFO: 'previous-logs' folder not found on ip-10-0-143-86.ec2.internal, skipping... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:04.173933258Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:04.173972831Z INFO: Machine Config Daemon termination log collection complete. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:04.200160548Z Wrote inspect data to must-gather. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:04.266146890Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:04.275562193Z error: the server doesn't have a resource type "machineconfigpools" [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:04.297904657Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:04.301027061Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:04.556504085Z INFO: Getting status/config from prometheus-k8s-0 [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:04.613514979Z W0727 18:31:04.613466 1205 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in '' [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:04.635490203Z Wrote inspect data to must-gather. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:04.704190643Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:04.741531367Z Wrote inspect data to must-gather. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:04.754930355Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:04.758562653Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:04.975533992Z Gathering data for ns/openshift-monitoring... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:05.064139789Z error: the server doesn't have a resource type "kubeletconfigs" [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:05.119283765Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:05.125648161Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:05.177624108Z INFO: INTERCONNECT MODE [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:05.177674200Z INFO: Gathering ovn-kubernetes DBs [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:05.198729036Z INFO: Getting status/flags from prometheus-k8s-0 [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:05.423990038Z INFO: Gathering OVN_Northbound from ovnkube-node-bd48h... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:05.424313205Z INFO: Gathering OVN_Northbound from ovnkube-node-grkdn... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:05.424609061Z INFO: Gathering OVN_Northbound from ovnkube-node-m9k9v... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:05.424933999Z INFO: Gathering OVN_Southbound from ovnkube-node-bd48h... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:05.426299919Z INFO: Gathering OVN_Southbound from ovnkube-node-grkdn... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:05.427290670Z INFO: Gathering OVN_Southbound from ovnkube-node-m9k9v... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:05.443770774Z Wrote inspect data to must-gather. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:05.461390136Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:05.469425329Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:05.627320289Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:05.788204690Z tar: Removing leading `/' from member names [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:05.788465343Z tar: Removing leading `/' from member names [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:05.810682395Z tar: Removing leading `/' from member names [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:05.825686636Z tar: Removing leading `/' from member names [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:05.848751461Z tar: Removing leading `/' from member names [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:05.889496197Z tar: Removing leading `/' from member names [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:05.971962817Z Wrote inspect data to must-gather. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:05.992063581Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:06.070142575Z INFO: Getting targets?state=active from prometheus-k8s-0 [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:06.313495964Z Wrote inspect data to must-gather. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:06.400729274Z INFO: Getting status/tsdb from prometheus-k8s-0 [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:06.462929739Z INFO: Waiting for network log collection to complete ... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:06.463055852Z INFO: Waiting for ovnk database copies to complete ... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:06.463978459Z INFO: Copying ovnk databases complete. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:06.466629850Z 6.6M must-gather/network_logs/ovnk_database_store [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:06.469331265Z ovnk_database_store/ [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:06.469331265Z ovnk_database_store/ovnkube-node-m9k9v_nbdb [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:06.484400360Z ovnk_database_store/ovnkube-node-bd48h_sbdb [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:06.529485227Z ovnk_database_store/ovnkube-node-grkdn_nbdb [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:06.547782353Z ovnk_database_store/ovnkube-node-bd48h_nbdb [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:06.567195448Z ovnk_database_store/ovnkube-node-m9k9v_sbdb [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:06.620754606Z ovnk_database_store/ovnkube-node-grkdn_sbdb [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:06.668618806Z INFO: Network log collection complete. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:06.680989159Z Wrote inspect data to must-gather. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:06.906168757Z INFO: Getting status from alertmanager-main-0 [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:06.987224543Z Wrote inspect data to must-gather. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:07.061728758Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:07.335686279Z Wrote inspect data to must-gather. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:07.752591210Z Gathering data for ns/openshift-network-console... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:07.766041042Z Wrote inspect data to must-gather. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:07.862188236Z Wrote inspect data to must-gather. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:08.076283449Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:08.078667437Z INFO: Fallback to identify the container image from release info [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:08.297674706Z Gathering data for ns/openshift-console-operator... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:08.447670358Z Inspecting resource crd/authorizationpolicies.security.istio.io [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:08.665908318Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:08.705874223Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811 [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:08.767128689Z Gathering data for ns/openshift-console... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:08.858575137Z daemonset.apps/perf-node-gather-daemonset created [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:08.941959120Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:08.943316266Z Inspecting resource crd/destinationrules.networking.istio.io [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:09.034443084Z Waiting for performance profile collector pods to become ready: 1 [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:09.203451479Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:09.454121853Z Gathering data for ns/openshift-cluster-storage-operator... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:09.511337961Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:09.516590055Z Inspecting resource crd/envoyfilters.networking.istio.io [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:09.725488129Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:09.986514011Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:09.988770973Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:10.086954517Z Gathering data for ns/openshift-dns-operator... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:10.213673739Z Waiting for performance profile collector pods to become ready: 2 [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:10.222509230Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:10.455679836Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:10.458108757Z Inspecting resource crd/gateways.gateway.networking.k8s.io [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:10.464798939Z Gathering data for ns/openshift-dns... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:10.645429443Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:10.869601815Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:10.872309652Z Inspecting resource crd/gateways.networking.istio.io [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:11.086594353Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:11.169286729Z Gathering data for ns/openshift-image-registry... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:11.317401210Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:11.320345208Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:11.368872801Z Waiting for performance profile collector pods to become ready: 3 [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:11.553501202Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:11.777330041Z Gathering data for ns/openshift-ingress-operator... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:11.791705681Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:11.794904437Z Inspecting resource crd/httproutes.gateway.networking.k8s.io [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:12.049198236Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:12.068545371Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:12.204615135Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:12.297447408Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:12.299973820Z Inspecting resource crd/inferencepools.inference.networking.k8s.io [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:12.481679790Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:12.511231756Z Waiting for performance profile collector pods to become ready: 4 [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:12.692650297Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:12.695154006Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:12.894505285Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:13.115336453Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:13.119442777Z Inspecting resource crd/peerauthentications.security.istio.io [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:13.237161336Z Gathering data for ns/openshift-ingress-canary... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:13.316718317Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:13.565388575Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:13.568933973Z Inspecting resource crd/proxyconfigs.networking.istio.io [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:13.684748013Z Waiting for performance profile collector pods to become ready: 5 [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:13.749516507Z Gathering data for ns/openshift-insights... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:13.783817777Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:13.986823111Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:13.989271835Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:14.184506725Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:14.424796451Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:14.427622549Z Inspecting resource crd/requestauthentications.security.istio.io [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:14.633015496Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:14.826291143Z Waiting for performance profile collector pods to become ready: 6 [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:14.868557557Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:14.871335586Z Inspecting resource crd/serviceentries.networking.istio.io [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:15.060100661Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:15.289935332Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:15.292486516Z Inspecting resource crd/sidecars.networking.istio.io [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:15.491331290Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:15.700029208Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:15.703519216Z Inspecting resource crd/telemetries.telemetry.istio.io [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:15.907303576Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:15.998372082Z Waiting for performance profile collector pods to become ready: 7 [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:16.129323230Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:16.131756981Z Inspecting resource crd/virtualservices.networking.istio.io [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:16.349643006Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:16.570803406Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:16.573297899Z Inspecting resource crd/wasmplugins.extensions.istio.io [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:16.771888915Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:17.008525634Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:17.012758101Z Inspecting resource crd/workloadentries.networking.istio.io [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:17.015161446Z Gathering data for ns/openshift-lws-operator... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:17.073790874Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:17.144521502Z Daemonset perf-node-gather-daemonset ready 3 out of 3 [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:17.242030843Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:17.506302401Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:17.509799559Z Inspecting resource crd/workloadgroups.networking.istio.io [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:17.515622298Z Collecting performance related data for node ip-10-0-132-194.ec2.internal [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:17.528972308Z Collecting performance related data for node ip-10-0-143-86.ec2.internal [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:17.541598953Z Collecting performance related data for node ip-10-0-137-134.ec2.internal [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:17.772051707Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:17.799514937Z Gathering data for ns/kserve... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:18.015426768Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:18.189463611Z Inspecting resource mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector-openshift-gateway-openshift-ingress [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:18.489808816Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:19.244956214Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:19.474738843Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:19.631639818Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:19.764615585Z Inspecting resource validatingwebhookconfiguration.admissionregistration.k8s.io/istio-validator-openshift-gateway-openshift-ingress [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:19.987492158Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:20.736891913Z Gathering data for ns/openshift-config... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:20.738996925Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:20.952790152Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:21.148476325Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:21.233387864Z Gathering data for ns/openshift-config-managed... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:21.654111220Z Done executing Istio gather script [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:21.750468516Z Gathering data for ns/openshift-kube-apiserver-operator... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:22.080531258Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:22.089383918Z Gathering data for ns/openshift-kube-apiserver... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:22.406163117Z Gathering data for ns/openshift-kube-controller-manager... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:22.774040318Z Gathering data for ns/openshift-kube-controller-manager-operator... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:23.138478758Z Gathering data for ns/openshift-kube-scheduler... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:23.467873225Z Gathering data for ns/openshift-kube-scheduler-operator... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:23.846818344Z Gathering data for ns/openshift-kube-storage-version-migrator... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:24.234064177Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:24.610108157Z Gathering data for ns/openshift-user-workload-monitoring... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:25.219528165Z Gathering data for ns/openshift-multus... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:26.681131103Z Collecting kubelet logs for node ip-10-0-132-194.ec2.internal [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:26.682949147Z Collecting kubelet logs for node ip-10-0-143-86.ec2.internal [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:26.684602204Z Collecting kubelet logs for node ip-10-0-137-134.ec2.internal [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:26.802395515Z Gathering data for ns/openshift-ovn-kubernetes... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:27.085366838Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:27.107520677Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-q5775 namespace [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:27.113180319Z INFO: Node performance data collection complete. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:28.307609158Z Gathering data for ns/openshift-host-network... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:28.687077965Z Gathering data for ns/openshift-network-diagnostics... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:29.270324623Z Gathering data for ns/openshift-network-node-identity... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:29.779641275Z Gathering data for ns/openshift-network-operator... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:30.204946858Z Gathering data for ns/openshift-cloud-network-config-controller... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:30.506701191Z Gathering data for ns/openshift-cluster-node-tuning-operator... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:31.037752242Z Gathering data for ns/openshift-apiserver-operator... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:31.390653353Z Gathering data for ns/openshift-apiserver... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:31.867237168Z Gathering data for ns/openshift-controller-manager-operator... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:32.090661713Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:32.196108453Z Gathering data for ns/openshift-controller-manager... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:32.507636446Z Gathering data for ns/openshift-cluster-samples-operator... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:33.226039011Z Gathering data for ns/openshift-operator-lifecycle-manager... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:33.549822599Z Gathering data for ns/openshift-service-ca-operator... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:33.898629012Z Gathering data for ns/openshift-service-ca... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:34.313171707Z Gathering data for ns/openshift-cluster-csi-drivers... [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:35.352681517Z Wrote inspect data to must-gather. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:35.352729652Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:35.352729652Z skipping gathering secrets/support due to error: secrets "support" not found [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:35.499629631Z error: the server doesn't have a resource type "clusters" [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:36.092350331Z Caches written to disk [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:35.352681517Z Wrote inspect data to must-gather. [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:35.352729652Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:35.352729652Z skipping gathering secrets/support due to error: secrets "support" not found [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:35.499629631Z error: the server doesn't have a resource type "clusters" [must-gather] [must-gather-dw5cm] POD 2026-07-27T18:31:36.092350331Z Caches written to disk [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.107443601Z waiting for gather to complete [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.109704564Z downloading gather output [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.359891317Z receiving incremental file list [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.382148022Z ./ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.382272805Z aggregated-discovery-api.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.382641604Z aggregated-discovery-apis.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.383425812Z event-filter.html [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.386189136Z timestamp [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.386263397Z version [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.393106506Z cluster-scoped-resources/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.393130806Z cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.393136936Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.39327987Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/default-network-annotation.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.393410773Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/servicecidrs.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.393535646Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/user-defined-networks-namespace-label.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.393593007Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.393678949Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/default-network-annotation-binding.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.393771321Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/servicecidrs-binding.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.393888194Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/user-defined-networks-namespace-label-binding.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.394029997Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.394086169Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/multus.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.394200321Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/network-node-identity.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.394269273Z cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.394723823Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.394759284Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/accounts.nim.opendatahub.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.394994899Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.395250705Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminpolicybasedexternalroutes.k8s.ovn.org.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.395618834Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertingrules.monitoring.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.395760887Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagerconfigs.monitoring.coreos.com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.3993403Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagers.monitoring.coreos.com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.400911407Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertrelabelconfigs.monitoring.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.40104988Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyapprovals.devportal.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.401164742Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyrequests.devportal.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.401275185Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeys.devportal.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.401400878Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiproducts.devportal.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.401883229Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apirequestcounts.apiserver.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.402057373Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiservers.config.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.402232187Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/appliedmanifestworks.work.open-cluster-management.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.40235869Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authconfigs.authorino.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.40323496Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authentications.config.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.403512947Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorinos.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.40364737Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.403837954Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authpolicies.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.405113224Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/baselineadminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.40536053Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/builds.config.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.405546054Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/bundles.trust.cert-manager.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.405744948Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/catalogsources.operators.coreos.com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.406087646Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificaterequests.cert-manager.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.40626061Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificates.cert-manager.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.406481345Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certmanagers.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.406788352Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/challenges.acme.cert-manager.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.40753281Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudcredentials.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.407651733Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.407766505Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudprivateipconfigs.cloud.network.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.407904578Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterclaims.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.408002191Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.408106853Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercsidrivers.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.408283457Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterimagepolicies.config.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.408460661Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterissuers.cert-manager.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.409225369Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteroperators.config.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.409354622Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterresourcequotas.quota.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.409477715Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterserviceversions.operators.coreos.com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.41141505Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterservingruntimes.serving.kserve.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.411986603Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterstoragecontainers.serving.kserve.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.412182088Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustertriggerauthentications.keda.sh.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.412367352Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteruserdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.412580227Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterversions.config.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.412840363Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.413306834Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.413421066Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.samples.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.413543939Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleclidownloads.console.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.413653551Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleexternalloglinks.console.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.413768164Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolelinks.console.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.413902157Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolenotifications.console.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.41401914Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleplugins.console.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.414176834Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolequickstarts.console.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.414324027Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.config.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.41443312Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.414696506Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolesamples.console.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.414816489Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleyamlsamples.console.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.414953842Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/controllerconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.415481234Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/credentialsrequests.cloudcredential.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.415613957Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/csisnapshotcontrollers.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.41572649Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/datascienceclusters.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.415931254Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.416656361Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.config.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.416754733Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.416966788Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnshealthcheckprobes.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.417075651Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnspolicies.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.417248145Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.ingress.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.417354677Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.417504551Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dscinitializations.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.417637454Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressfirewalls.k8s.ovn.org.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.417750246Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressips.k8s.ovn.org.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.41788824Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressqoses.k8s.ovn.org.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.418061824Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressrouters.network.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.418188907Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressservices.k8s.ovn.org.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.418293609Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.418447013Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/etcds.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.418574996Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/featuregates.config.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.418704799Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.418849642Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.419255801Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.419423965Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.419810744Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/helmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.419971418Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.420714975Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httpscaledobjects.http.keda.sh.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.420848648Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentpolicies.config.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.421014582Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentsourcepolicies.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.421121424Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagedigestmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.421241087Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepolicies.config.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.421437002Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepruners.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.421681717Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/images.config.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.421814371Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagetagmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.421949154Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencegraphs.serving.kserve.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.422102727Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.llm-d.ai.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.42223983Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.llm-d.ai.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.422354533Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.422495726Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.422620899Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.426460278Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/infrastructures.config.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.42696334Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresscontrollers.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.427517552Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresses.config.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.427701167Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/insightsoperators.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.42785196Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/installplans.operators.coreos.com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.428055315Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/interceptorroutes.http.keda.sh.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.428233109Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ipamclaims.k8s.cni.cncf.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.428324341Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ippools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.428486405Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/issuers.cert-manager.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.429276233Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiocsrs.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.430528382Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kedacontrollers.keda.sh.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.434030993Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kuadrants.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.434181227Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeapiservers.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.43433915Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubecontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.434489014Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeschedulers.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.434615047Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubestorageversionmigrators.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.43474794Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersetoperators.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.434882203Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersets.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.444965487Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/limitadors.limitador.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.445225053Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceserviceconfigs.serving.kserve.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.450372532Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.455329587Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfignodes.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.455504691Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfigurations.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.455782807Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosbuilds.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.456131855Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.456154056Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/network-attachment-definitions.k8s.cni.cncf.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.456219277Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.config.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.456369011Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.456627917Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodes.config.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.456747519Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodeslicepools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.456846102Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oauths.config.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.457068997Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oidcpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.45719734Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/olmconfigs.operators.coreos.com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.457304172Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftapiservers.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.457422215Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftcontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.457530778Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorconditions.operators.coreos.com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.45766181Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorgroups.operators.coreos.com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.457810524Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorhubs.config.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.457953407Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorpkis.network.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.458098301Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operators.operators.coreos.com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.458218454Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/orders.acme.cert-manager.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.458349477Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/overlappingrangeipreservations.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.458460239Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.458592512Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/pinnedimagesets.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.458713985Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/planpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.458825868Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podmonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.459161525Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podnetworkconnectivitychecks.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.459283238Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/probes.monitoring.coreos.com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.459539414Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/profiles.tuned.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.459678697Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projecthelmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.459824561Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projects.config.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.459976514Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheuses.monitoring.coreos.com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.462018251Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheusrules.monitoring.coreos.com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.462239736Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxies.config.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.462256637Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.462359839Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rangeallocations.security.internal.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.462465482Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.462612285Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.462734538Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.462913082Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rolebindingrestrictions.authorization.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.463034655Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledjobs.keda.sh.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.464634132Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledobjects.keda.sh.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.464801936Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/schedulers.config.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.46496151Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/securitycontextconstraints.security.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.465116123Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicecas.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.465233726Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.46541025Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicemonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.465663736Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servingruntimes.serving.kserve.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.466172908Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.466432674Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storages.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.466549816Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storagestates.migration.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.466672269Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storageversionmigrations.migration.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.466826123Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/subscriptions.operators.coreos.com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.467478958Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.467659202Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetrypolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.467772345Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/thanosrulers.monitoring.coreos.com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.469514685Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tlspolicies.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.46971445Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tokenratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.469915924Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trainedmodels.serving.kserve.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.470049377Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/triggerauthentications.keda.sh.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.470252032Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trustmanagers.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.470551519Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tuneds.tuned.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.470738843Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/userdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.470937228Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.471322927Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumepopulators.populator.storage.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.471420979Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotclasses.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.471534152Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotcontents.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.471747677Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshots.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.471934591Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.472062844Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.472199307Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.472366061Z cluster-scoped-resources/apiregistration.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.472603807Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.472656588Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1..yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.47274383Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.acme.cert-manager.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.472835502Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.472939674Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.473022516Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiserver.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.47317857Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.473281512Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.473376774Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.473466887Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.473544078Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.473639391Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.autoscaling.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.473740643Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.batch.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.473822225Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.build.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.473948228Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cert-manager.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.474027169Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.474116022Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloud.network.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.474201113Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloudcredential.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.474286746Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.config.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.474371628Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.console.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.47445987Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.474541501Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.474642734Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.discovery.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.474734806Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.474830838Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.events.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.474942781Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.flowcontrol.apiserver.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.475026172Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.475117335Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.image.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.475200367Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.475289459Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.inference.networking.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.475381161Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.ingress.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.475461033Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.475547955Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.ovn.org.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.475636437Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.475720779Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.475823441Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.475954894Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.coreos.com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.476041236Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.476126168Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.network.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.47621432Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.476312102Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.476400924Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.nim.opendatahub.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.476498007Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.node.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.476582179Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.oauth.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.476669611Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.476753263Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operators.coreos.com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.476840234Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.packages.operators.coreos.com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.477016329Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.policy.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.477134301Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.project.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.477254594Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.quota.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.477346436Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.477434468Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.resource.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.477526821Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.route.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.477626513Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.samples.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.477703345Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.scheduling.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.477810217Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.internal.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.47792794Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.478021672Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.478115384Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.478207686Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.storage.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.478293308Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.telemetry.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.47837865Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.template.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.478470552Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.tuned.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.478558324Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.user.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.478667247Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.work.open-cluster-management.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.47877769Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.478867561Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.478985674Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.devportal.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.479079937Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.eventing.keda.sh.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.479169919Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.479276701Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.479349493Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.http.keda.sh.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.479446875Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.479545097Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.keda.sh.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.479637579Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.479735002Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.limitador.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.479887635Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.migration.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.479973717Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.monitoring.coreos.com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.48008266Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operator.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.480187342Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operators.coreos.com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.480293685Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.policy.networking.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.480633842Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.serving.kserve.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.480655073Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.telemetry.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.480662783Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.trust.cert-manager.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.480701994Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.480806196Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.480931949Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.llm-d.ai.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.481032242Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.operators.coreos.com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.481140254Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.serving.kserve.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.481235056Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha3.networking.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.481347029Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.481452701Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.external.metrics.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.481581884Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.481685227Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.helm.openshift.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.48179914Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.http.keda.sh.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.481922432Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.482023975Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.metrics.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.482142087Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.monitoring.coreos.com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.48224768Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.networking.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.482358152Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.482476825Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.populator.storage.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.482572597Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.security.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.48271115Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.serving.kserve.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.482816133Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.storage.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.483004267Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta3.authorino.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.48311053Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.autoscaling.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.48314026Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.operators.coreos.com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.483218572Z cluster-scoped-resources/certificates.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.483234863Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.483287044Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-f5qb8.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.483401406Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-jw54s.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.483507219Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-nd697.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.483737794Z cluster-scoped-resources/config.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.483794056Z cluster-scoped-resources/config.openshift.io/apiservers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.484015511Z cluster-scoped-resources/config.openshift.io/authentications.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.484193405Z cluster-scoped-resources/config.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.484288147Z cluster-scoped-resources/config.openshift.io/clusterimagepolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.4843989Z cluster-scoped-resources/config.openshift.io/clusteroperators.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.484780658Z cluster-scoped-resources/config.openshift.io/clusterversions.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.484989943Z cluster-scoped-resources/config.openshift.io/consoles.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.485119496Z cluster-scoped-resources/config.openshift.io/dnses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.485233399Z cluster-scoped-resources/config.openshift.io/featuregates.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.485359962Z cluster-scoped-resources/config.openshift.io/imagecontentpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.485460794Z cluster-scoped-resources/config.openshift.io/imagedigestmirrorsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.485563837Z cluster-scoped-resources/config.openshift.io/imagepolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.485641829Z cluster-scoped-resources/config.openshift.io/images.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.48572718Z cluster-scoped-resources/config.openshift.io/imagetagmirrorsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.485817542Z cluster-scoped-resources/config.openshift.io/infrastructures.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.485935105Z cluster-scoped-resources/config.openshift.io/ingresses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.486024057Z cluster-scoped-resources/config.openshift.io/networks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.486108729Z cluster-scoped-resources/config.openshift.io/nodes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.486198141Z cluster-scoped-resources/config.openshift.io/oauths.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.486280503Z cluster-scoped-resources/config.openshift.io/operatorhubs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.486408436Z cluster-scoped-resources/config.openshift.io/projects.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.486513278Z cluster-scoped-resources/config.openshift.io/proxies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.486661192Z cluster-scoped-resources/config.openshift.io/schedulers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.486779265Z cluster-scoped-resources/config.openshift.io/clusteroperators/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.486831946Z cluster-scoped-resources/config.openshift.io/clusteroperators/console.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.486975179Z cluster-scoped-resources/config.openshift.io/clusteroperators/csi-snapshot-controller.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.487070122Z cluster-scoped-resources/config.openshift.io/clusteroperators/dns.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.487186874Z cluster-scoped-resources/config.openshift.io/clusteroperators/image-registry.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.487311747Z cluster-scoped-resources/config.openshift.io/clusteroperators/ingress.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.487419879Z cluster-scoped-resources/config.openshift.io/clusteroperators/insights.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.487562963Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-apiserver.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.487724777Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-controller-manager.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.487841389Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-scheduler.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.487974243Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-storage-version-migrator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.488085755Z cluster-scoped-resources/config.openshift.io/clusteroperators/monitoring.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.488195277Z cluster-scoped-resources/config.openshift.io/clusteroperators/network.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.488337311Z cluster-scoped-resources/config.openshift.io/clusteroperators/node-tuning.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.488444203Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-apiserver.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.488557346Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-controller-manager.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.488667168Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-samples.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.488775501Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-catalog.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.488906834Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-packageserver.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.489013616Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.489125329Z cluster-scoped-resources/config.openshift.io/clusteroperators/service-ca.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.489239802Z cluster-scoped-resources/config.openshift.io/clusteroperators/storage.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.489330714Z cluster-scoped-resources/config.openshift.io/clusterversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.489389555Z cluster-scoped-resources/config.openshift.io/clusterversions/version.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.489469827Z cluster-scoped-resources/config.openshift.io/consoles/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.489523008Z cluster-scoped-resources/config.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.48959878Z cluster-scoped-resources/config.openshift.io/featuregates/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.489692142Z cluster-scoped-resources/config.openshift.io/featuregates/cluster.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.489786624Z cluster-scoped-resources/config.openshift.io/infrastructures/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.489840576Z cluster-scoped-resources/config.openshift.io/infrastructures/cluster.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.490010579Z cluster-scoped-resources/config.openshift.io/oauths/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.490097061Z cluster-scoped-resources/config.openshift.io/oauths/cluster.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.490186124Z cluster-scoped-resources/config.openshift.io/proxies/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.490209654Z cluster-scoped-resources/config.openshift.io/proxies/cluster.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.490288046Z cluster-scoped-resources/console.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.490302346Z cluster-scoped-resources/console.openshift.io/consoleplugins/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.490370438Z cluster-scoped-resources/console.openshift.io/consoleplugins/kuadrant-console-plugin.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.4904765Z cluster-scoped-resources/console.openshift.io/consoleplugins/monitoring-plugin.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.490579973Z cluster-scoped-resources/console.openshift.io/consoleplugins/networking-console-plugin.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.490692075Z cluster-scoped-resources/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.490701315Z cluster-scoped-resources/core/nodes/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.490760997Z cluster-scoped-resources/core/nodes/ip-10-0-132-194.ec2.internal.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.490994002Z cluster-scoped-resources/core/nodes/ip-10-0-137-134.ec2.internal.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.491145626Z cluster-scoped-resources/core/nodes/ip-10-0-143-86.ec2.internal.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.491258568Z cluster-scoped-resources/core/persistentvolumes/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.491297759Z cluster-scoped-resources/core/persistentvolumes/pvc-56638b18-3f61-49c9-8c07-2721db06574b.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.491394992Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.491459023Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.491521514Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/catch-all.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.491632407Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/exempt.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.491727349Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/global-default.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.491816341Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-controller-manager.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.491932664Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-scheduler.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.492009446Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-system-service-accounts.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.49217148Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.492266852Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-sar.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.492354454Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.492448316Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-authentication-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.492560209Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-controller-manager.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.492685461Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.492774054Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-monitoring-metrics.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.492876696Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver-sar.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.492987479Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.493096251Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-server.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.493194143Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/probes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.493311056Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/service-accounts.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.493404278Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-leader-election.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.49349469Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-node-high.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.493585582Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-nodes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.493702035Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.493735216Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/catch-all.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.493843068Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/exempt.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.493976271Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/global-default.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.494112785Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/leader-election.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.494214797Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/node-high.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.494312459Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/openshift-control-plane-operators.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.494543744Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/system.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.494710288Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-high.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.494808231Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-low.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.494890572Z cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.494910543Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.494934053Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.495000395Z cluster-scoped-resources/imageregistry.operator.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.495016446Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.495032736Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.495134778Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.495178139Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/cluster.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.495247511Z cluster-scoped-resources/migration.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.495257711Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.495320482Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/console-plugin-storage-version-migration.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.495401304Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-controllerconfig-storage-version-migration.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.495556638Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-machineconfigpool-storage-version-migration.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.495612739Z cluster-scoped-resources/oauth.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.4956204Z cluster-scoped-resources/oauth.openshift.io/oauthclients/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.495669931Z cluster-scoped-resources/oauth.openshift.io/oauthclients/console.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.495790704Z cluster-scoped-resources/operator.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.495821404Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.495832764Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.495995308Z cluster-scoped-resources/operator.openshift.io/consoles/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.496038849Z cluster-scoped-resources/operator.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.496401727Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.496437628Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/cluster.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.49651123Z cluster-scoped-resources/operator.openshift.io/dnses/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.496549371Z cluster-scoped-resources/operator.openshift.io/dnses/default.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.496609612Z cluster-scoped-resources/operator.openshift.io/insightsoperators/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.496685634Z cluster-scoped-resources/operator.openshift.io/insightsoperators/cluster.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.496812357Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.496850528Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/cluster.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.49695148Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.496989871Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/cluster.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.497061543Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.497088883Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/cluster.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.497148705Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.497205546Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/cluster.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.497263827Z cluster-scoped-resources/operator.openshift.io/networks/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.497302068Z cluster-scoped-resources/operator.openshift.io/networks/cluster.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.49737956Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.497409561Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/cluster.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.497471172Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.497515003Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/cluster.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.497574435Z cluster-scoped-resources/operator.openshift.io/servicecas/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.497613776Z cluster-scoped-resources/operator.openshift.io/servicecas/cluster.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.497682377Z cluster-scoped-resources/operator.openshift.io/storages/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.497720048Z cluster-scoped-resources/operator.openshift.io/storages/cluster.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.497828791Z cluster-scoped-resources/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.497836481Z cluster-scoped-resources/operators.coreos.com/olmconfigs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.497892052Z cluster-scoped-resources/operators.coreos.com/olmconfigs/cluster.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.498015305Z cluster-scoped-resources/operators.coreos.com/operators/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.498061346Z cluster-scoped-resources/operators.coreos.com/operators/authorino-operator.kuadrant-system.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.498169519Z cluster-scoped-resources/operators.coreos.com/operators/dns-operator.kuadrant-system.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.498267641Z cluster-scoped-resources/operators.coreos.com/operators/leader-worker-set.openshift-lws-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.498450545Z cluster-scoped-resources/operators.coreos.com/operators/limitador-operator.kuadrant-system.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.498556528Z cluster-scoped-resources/operators.coreos.com/operators/openshift-cert-manager-operator.cert-manager-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.498695761Z cluster-scoped-resources/operators.coreos.com/operators/openshift-custom-metrics-autoscaler-operator.openshift-keda.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.498812673Z cluster-scoped-resources/operators.coreos.com/operators/rhcl-operator.kuadrant-system.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.498929456Z cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.498950177Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.499001908Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/aws-ebs-csi-driver-operator-clusterrolebinding.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.49910603Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cloud-network-config-controller.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.499239023Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-node-tuning-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.499382147Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-storage-operator-role.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.49952884Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.499620542Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-runner-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.499749205Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/metrics-daemon-sa-rolebinding.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.499840117Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-admission-controller-webhook.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.49998625Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-ancillary-tools.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.500079943Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-cluster-readers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.500174665Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-group.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.500259117Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-transient.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.500350529Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-whereabouts.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.500441281Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-diagnostics.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.500534373Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-node-identity.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.500601445Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-image-registry-pruner.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.500693867Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-iptables-alerter.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.500780969Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.500894462Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-identity-limited.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.500989224Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-kube-rbac-proxy.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.501086266Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/registry-registry-role.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.501219159Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.50126166Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/aws-ebs-csi-driver-operator-clusterrole.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.501367033Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cloud-network-config-controller.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.501456235Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cluster-node-tuning-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.501544847Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-extensions-reader.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.50170448Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.501793123Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.501932666Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.502031098Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/helm-chartrepos-viewer.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.50212536Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/metrics-daemon-role.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.502223972Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-admission-controller-webhook.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.502336315Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-ancillary-tools.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.502452508Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.50257176Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/net-attach-def-project.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.502696953Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-diagnostics.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.502806086Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-node-identity.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.502932959Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-csi-snapshot-controller-runner.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.503033551Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-iptables-alerter.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.503135294Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-cluster-reader.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.503289287Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.503381249Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-kube-rbac-proxy.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.503471351Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.503547503Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-editor.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.503634695Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-viewer.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.503725717Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/project-helm-chartrepository-editor.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.503810789Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-admin.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.503933392Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-basic-user.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.504018084Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-storage-admin.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.504103446Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-view.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.504193218Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:registry.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.50429038Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/whereabouts-cni.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.504336771Z cluster-scoped-resources/samples.operator.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.504344742Z cluster-scoped-resources/samples.operator.openshift.io/configs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.504384752Z cluster-scoped-resources/samples.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.504431704Z cluster-scoped-resources/snapshot.storage.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.504448644Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.504495495Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/csi-aws-vsc.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.504569697Z cluster-scoped-resources/storage.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.504577637Z cluster-scoped-resources/storage.k8s.io/csidrivers/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.504616538Z cluster-scoped-resources/storage.k8s.io/csidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.50469005Z cluster-scoped-resources/storage.k8s.io/csinodes/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.50472463Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-132-194.ec2.internal.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.504815742Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-137-134.ec2.internal.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.504916745Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-143-86.ec2.internal.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.504984646Z cluster-scoped-resources/storage.k8s.io/storageclasses/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.505028107Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp2-csi.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.50513004Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp3-csi.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.505181621Z cluster-scoped-resources/storage.k8s.io/volumeattachments/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.505222942Z cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-cf909ba9ccf0434be837965c3d491f51f0b6187d703013bb433c17cc3db357d5.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.505293413Z host_service_logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.505331684Z host_service_logs/masters/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.505373255Z host_service_logs/masters/NetworkManager_service.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.505444267Z host_service_logs/masters/crio_service.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.505516298Z host_service_logs/masters/kubelet_service.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.50556978Z host_service_logs/masters/machine-config-daemon-firstboot_service.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.505694993Z host_service_logs/masters/machine-config-daemon-host_service.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.505804305Z host_service_logs/masters/openvswitch_service.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.505897927Z host_service_logs/masters/ostree-finalize-staged_service.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.505970649Z host_service_logs/masters/ovs-configuration_service.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.506049531Z host_service_logs/masters/ovs-vswitchd_service.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.506100612Z host_service_logs/masters/ovsdb-server_service.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.506176894Z host_service_logs/masters/rpm-ostreed_service.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.506217815Z ingress_controllers/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.506227865Z ingress_controllers/default/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.506231855Z ingress_controllers/default/router-default-57898f8749-pb7sc/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.506276776Z ingress_controllers/default/router-default-57898f8749-pb7sc/haproxy.config [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.506470981Z insights-data/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.506504021Z istio/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.506563833Z istio/aggregated-discovery-api.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.506702306Z istio/aggregated-discovery-apis.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.507236929Z istio/event-filter.html [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.507425003Z istio/timestamp [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.507532435Z istio/cluster-scoped-resources/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.507551076Z istio/cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.507557786Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.507573626Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/istio-sidecar-injector-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.507670248Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.507703419Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/istio-validator-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.507773471Z istio/cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.507827102Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.507924234Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.508079658Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.508929268Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.509222164Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.509413779Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.509816078Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.51035149Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.510794681Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.51161662Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.511757433Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.511909467Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.512036969Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.512144122Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.512266925Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.512545271Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.512747186Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.513038763Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.513251388Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.513664967Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.51379541Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.513986845Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.514170409Z istio/cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.514179039Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.51421141Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.514274741Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.514283761Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.514341183Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.514427895Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.514573188Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.51465617Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.514707561Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.514804674Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.514997698Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.51507537Z istio/namespaces/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.51508465Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.515091Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/gateway.networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.515100511Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.515149172Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/gateway.networking.k8s.io/httproutes/fc-concurrency-test-kserve-route.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.515230014Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/inference.networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.515237964Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.515283195Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/inference.networking.k8s.io/inferencepools/fc-concurrency-test-inference-pool.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.515362606Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/inference.networking.x-k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.515370517Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.515427988Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/inference.networking.x-k8s.io/inferencepools/fc-concurrency-test-inference-pool.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.5155023Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/networking.istio.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.51551015Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/networking.istio.io/destinationrules/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.515542711Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/networking.istio.io/destinationrules/fc-concurrency-test-kserve-scheduler.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.515635583Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/networking.istio.io/destinationrules/fc-concurrency-test-kserve-shadow-svc.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.515729935Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/networking.istio.io/destinationrules/fc-concurrency-test-kserve-workload-svc.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.515820357Z istio/namespaces/e2e-test-llm-inference-service-48639af5/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.515828577Z istio/namespaces/e2e-test-llm-inference-service-48639af5/gateway.networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.515838888Z istio/namespaces/e2e-test-llm-inference-service-48639af5/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.515912849Z istio/namespaces/e2e-test-llm-inference-service-48639af5/gateway.networking.k8s.io/httproutes/llmisvc-model-pvc-router-manage-e8706282-kserve-route.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.516029682Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.516041852Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.516085153Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.k8s.io/inferencepools/llmisvc-model-pvc-router-manage-e8706282-inference-pool.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.516163385Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.x-k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.516170865Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.516215776Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.x-k8s.io/inferencepools/llmisvc-model-pvc-router-manage-e8706282-inference-pool.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.516282278Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.516295398Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.516349469Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.516457542Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.516618636Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.516700708Z istio/namespaces/e2e-test-llm-inference-service-fac1c4ae/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.516719598Z istio/namespaces/e2e-test-llm-inference-service-fac1c4ae/gateway.networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.516725308Z istio/namespaces/e2e-test-llm-inference-service-fac1c4ae/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.516745189Z istio/namespaces/e2e-test-llm-inference-service-fac1c4ae/gateway.networking.k8s.io/httproutes/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.51682488Z istio/namespaces/e2e-test-llm-inference-service-fac1c4ae/networking.istio.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.516835661Z istio/namespaces/e2e-test-llm-inference-service-fac1c4ae/networking.istio.io/destinationrules/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.516922753Z istio/namespaces/e2e-test-llm-inference-service-fac1c4ae/networking.istio.io/destinationrules/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.516987624Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.516995744Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/gateway.networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.517005035Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.517060676Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/gateway.networking.k8s.io/httproutes/stop-feature-test-kserve-route.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.517166158Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/inference.networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.517180839Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.517189749Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/inference.networking.k8s.io/inferencepools/stop-feature-test-inference-pool.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.51724228Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/inference.networking.x-k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.51725277Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.517308182Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/inference.networking.x-k8s.io/inferencepools/stop-feature-test-inference-pool.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.517364183Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/networking.istio.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.517373023Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/networking.istio.io/destinationrules/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.517408334Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/networking.istio.io/destinationrules/stop-feature-test-kserve-scheduler.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.517490386Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/networking.istio.io/destinationrules/stop-feature-test-kserve-shadow-svc.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.517579318Z istio/namespaces/e2e-test-llm-stop-feature-d9eef2a1/networking.istio.io/destinationrules/stop-feature-test-kserve-workload-svc.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.517749612Z istio/namespaces/openshift-ingress/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.517783743Z istio/namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.517882555Z istio/namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.518013208Z istio/namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.51808146Z istio/namespaces/openshift-ingress/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.518130051Z istio/namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.518212072Z istio/namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.518622722Z istio/namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.518876418Z istio/namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.518939429Z istio/namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.518996211Z istio/namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.519084393Z istio/namespaces/openshift-ingress/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.519115673Z istio/namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.519205535Z istio/namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.519267677Z istio/namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.519316458Z istio/namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.51939626Z istio/namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.519473892Z istio/namespaces/openshift-ingress/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.519518813Z istio/namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.519983604Z istio/namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.520079656Z istio/namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.520299531Z istio/namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.520377443Z istio/namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.520623628Z istio/namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.520776652Z istio/namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.521024158Z istio/namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.52112519Z istio/namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.521152941Z istio/namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.521257163Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.521266323Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.521306084Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.521385326Z istio/namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.521427707Z istio/namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.521491448Z istio/namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.52154472Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.521632812Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.521718484Z istio/namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.521762455Z istio/namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.521811636Z istio/namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.521875857Z istio/namespaces/openshift-ingress/networking.istio.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.521927879Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.52198869Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-auth-openshift-ai-inference.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.522109393Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-openshift-ai-inference.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.522309058Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/openshift-ai-inference-authn-ssl.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.52241154Z istio/namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.522467081Z istio/namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.522538563Z istio/namespaces/openshift-ingress/pods/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.522545743Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-hkbqw/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.522591624Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-hkbqw/istiod-openshift-gateway-5677ccb49b-hkbqw.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.522725727Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-hkbqw/discovery/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.522734567Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-hkbqw/discovery/discovery/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.522741407Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-hkbqw/discovery/discovery/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.522786438Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-hkbqw/discovery/discovery/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.532393101Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-hkbqw/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.532442432Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-hkbqw/discovery/discovery/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.532507463Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.532548304Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.532673317Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c/istio-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.532681867Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c/istio-proxy/istio-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.532686908Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.532728739Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.534030269Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.534135641Z istio/namespaces/openshift-ingress/pods/router-default-57898f8749-pb7sc/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.534186902Z istio/namespaces/openshift-ingress/pods/router-default-57898f8749-pb7sc/router-default-57898f8749-pb7sc.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.534297695Z istio/namespaces/openshift-ingress/pods/router-default-57898f8749-pb7sc/router/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.534306305Z istio/namespaces/openshift-ingress/pods/router-default-57898f8749-pb7sc/router/router/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.534315616Z istio/namespaces/openshift-ingress/pods/router-default-57898f8749-pb7sc/router/router/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.534389947Z istio/namespaces/openshift-ingress/pods/router-default-57898f8749-pb7sc/router/router/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.534501Z istio/namespaces/openshift-ingress/pods/router-default-57898f8749-pb7sc/router/router/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.534592052Z istio/namespaces/openshift-ingress/pods/router-default-57898f8749-pb7sc/router/router/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.534611012Z istio/namespaces/openshift-ingress/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.534683854Z istio/namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.534739405Z istio/namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.534759656Z istio/namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.534828347Z machine_config_ondisk/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.534837448Z machine_config_termination_logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.534845308Z monitoring/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.534852668Z monitoring/alertmanager/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.53494746Z monitoring/alertmanager/status.json [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.535026902Z monitoring/alertmanager/status.stderr [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.535084333Z monitoring/prometheus/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.535133734Z monitoring/prometheus/alertmanagers.json [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.535220246Z monitoring/prometheus/alertmanagers.stderr [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.535285638Z monitoring/prometheus/rules.json [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.536001624Z monitoring/prometheus/rules.stderr [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.536061466Z monitoring/prometheus/prometheus-k8s-0/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.536103047Z monitoring/prometheus/prometheus-k8s-0/active-targets.json [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.53709478Z monitoring/prometheus/prometheus-k8s-0/active-targets.stderr [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.537151691Z monitoring/prometheus/prometheus-k8s-0/status/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.537195422Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.json [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.537291984Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.stderr [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.537357656Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.json [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.537449018Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.stderr [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.53751745Z monitoring/prometheus/status/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.537559731Z monitoring/prometheus/status/config.json [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.538029622Z monitoring/prometheus/status/config.stderr [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.538091493Z monitoring/prometheus/status/flags.json [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.538189495Z monitoring/prometheus/status/flags.stderr [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.538438361Z namespaces/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.538447561Z namespaces/cert-manager-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.538457371Z namespaces/cert-manager-operator/coordination.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.538463372Z namespaces/cert-manager-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.538504572Z namespaces/cert-manager-operator/coordination.k8s.io/leases/cert-manager-operator-lock.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.538604315Z namespaces/cert-manager-operator/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.538614515Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.538680757Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.538921762Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.539232589Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.539375602Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.539591387Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.539791952Z namespaces/cert-manager-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.539809733Z namespaces/cert-manager-operator/operators.coreos.com/installplans/install-2q54k.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.540003337Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.540036898Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.540098029Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.54014411Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/openshift-cert-manager-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.540200592Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.540239322Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/openshift-cert-manager-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.540297874Z namespaces/cert-manager/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.540305244Z namespaces/cert-manager/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.540309364Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.540340855Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.540495279Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.540670732Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.540804626Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.541035931Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.541254136Z namespaces/default/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.541302087Z namespaces/default/default.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.541388799Z namespaces/default/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.54143906Z namespaces/default/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.541492362Z namespaces/default/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.541522122Z namespaces/default/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.541599144Z namespaces/default/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.541686636Z namespaces/default/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.541765168Z namespaces/default/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.541823279Z namespaces/default/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.54185284Z namespaces/default/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.541953022Z namespaces/default/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.542001273Z namespaces/default/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.542085625Z namespaces/default/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.542144447Z namespaces/default/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.542193018Z namespaces/default/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.54227385Z namespaces/default/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.542350331Z namespaces/default/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.542404463Z namespaces/default/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.542508985Z namespaces/default/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.542594607Z namespaces/default/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.542771011Z namespaces/default/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.542846963Z namespaces/default/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.542948475Z namespaces/default/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.543068408Z namespaces/default/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.543220611Z namespaces/default/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.543330184Z namespaces/default/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.543373325Z namespaces/default/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.543440147Z namespaces/default/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.543663812Z namespaces/default/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.543672802Z namespaces/default/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.543676732Z namespaces/default/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.543680522Z namespaces/default/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.543686893Z namespaces/default/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.543762444Z namespaces/default/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.543802645Z namespaces/default/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.543877267Z namespaces/default/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.543929558Z namespaces/default/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.543976669Z namespaces/default/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.54399454Z namespaces/default/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.544050811Z namespaces/default/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.544201494Z namespaces/default/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.544386478Z namespaces/default/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.544556873Z namespaces/default/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.544747557Z namespaces/default/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.544978342Z namespaces/default/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.545013553Z namespaces/default/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.545104245Z namespaces/default/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.545127806Z namespaces/default/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.545209828Z namespaces/e2e-test-flow-control-smoke-1e98f46d/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.545220568Z namespaces/e2e-test-flow-control-smoke-1e98f46d/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.545225568Z namespaces/e2e-test-flow-control-smoke-1e98f46d/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.545266489Z namespaces/e2e-test-flow-control-smoke-1e98f46d/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine-default.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.545365991Z namespaces/e2e-test-flow-control-smoke-1e98f46d/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.545436383Z namespaces/e2e-test-flow-control-smoke-1e98f46d/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.545460514Z namespaces/e2e-test-flow-control-smoke-1e98f46d/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.545567806Z namespaces/e2e-test-flow-control-smoke-1e98f46d/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.545614897Z namespaces/e2e-test-flow-control-smoke-1e98f46d/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.545623547Z namespaces/e2e-test-flow-control-smoke-1e98f46d/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.545665298Z namespaces/e2e-test-flow-control-smoke-1e98f46d/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.545807391Z namespaces/e2e-test-flow-control-smoke-1e98f46d/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.546006316Z namespaces/e2e-test-flow-control-smoke-1e98f46d/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.546140909Z namespaces/e2e-test-flow-control-smoke-1e98f46d/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.546335284Z namespaces/e2e-test-flow-control-smoke-1e98f46d/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.546516208Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.546524188Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.546528568Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.546559769Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.546738263Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.546934837Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.547068091Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.547257405Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.547529761Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.547537831Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.547544192Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.547596003Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.547756007Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.547987792Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.548126745Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.54835138Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.548543355Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.548554835Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.548566865Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.548642907Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.548794631Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.549031346Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.549159539Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.549347833Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.549545958Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.549555998Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.549561418Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.549599909Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.549747723Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.549968798Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.550107591Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.550296126Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.550475439Z namespaces/e2e-test-llm-inference-service-48639af5/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.5504878Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.55049663Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.550535201Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine-default.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.550624303Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.550711915Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.550737606Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.550844328Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.55091012Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.55093698Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.550971441Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.551119974Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.551299999Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.551435402Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.551632376Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.551829961Z namespaces/e2e-test-llm-inference-service-fac1c4ae/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.551838441Z namespaces/e2e-test-llm-inference-service-fac1c4ae/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.551846951Z namespaces/e2e-test-llm-inference-service-fac1c4ae/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.551920343Z namespaces/e2e-test-llm-inference-service-fac1c4ae/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine-default.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.552005235Z namespaces/e2e-test-llm-inference-service-fac1c4ae/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.552057156Z namespaces/e2e-test-llm-inference-service-fac1c4ae/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.552111088Z namespaces/e2e-test-llm-inference-service-fac1c4ae/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.55220906Z namespaces/e2e-test-llm-inference-service-fac1c4ae/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.552286921Z namespaces/e2e-test-llm-inference-service-fac1c4ae/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.552321042Z namespaces/e2e-test-llm-inference-service-fac1c4ae/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.552364483Z namespaces/e2e-test-llm-inference-service-fac1c4ae/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.552517327Z namespaces/e2e-test-llm-inference-service-fac1c4ae/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.552724672Z namespaces/e2e-test-llm-inference-service-fac1c4ae/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.552850414Z namespaces/e2e-test-llm-inference-service-fac1c4ae/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.55309292Z namespaces/e2e-test-llm-inference-service-fac1c4ae/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.553293415Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.553303795Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.553317185Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.553422438Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine-default.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.55352564Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.553586682Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.553619272Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.553718945Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.553784706Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.553792416Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.553831997Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.554002701Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.554183716Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.554320309Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.554512383Z namespaces/e2e-test-llm-stop-feature-d9eef2a1/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.554706157Z namespaces/e2e-test-llm-with-lora-adapters-3cbf5e9c/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.554714898Z namespaces/e2e-test-llm-with-lora-adapters-3cbf5e9c/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.554718868Z namespaces/e2e-test-llm-with-lora-adapters-3cbf5e9c/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.554743068Z namespaces/e2e-test-llm-with-lora-adapters-3cbf5e9c/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.554920693Z namespaces/e2e-test-llm-with-lora-adapters-3cbf5e9c/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.555099967Z namespaces/e2e-test-llm-with-lora-adapters-3cbf5e9c/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.55523636Z namespaces/e2e-test-llm-with-lora-adapters-3cbf5e9c/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.555430854Z namespaces/e2e-test-llm-with-lora-adapters-3cbf5e9c/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.555610018Z namespaces/e2e-test-llm-with-lora-adapters-bbeb2681/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.55568925Z namespaces/e2e-test-llm-with-lora-adapters-bbeb2681/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.555702751Z namespaces/e2e-test-llm-with-lora-adapters-bbeb2681/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.555756252Z namespaces/e2e-test-llm-with-lora-adapters-bbeb2681/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.555934306Z namespaces/e2e-test-llm-with-lora-adapters-bbeb2681/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.55611181Z namespaces/e2e-test-llm-with-lora-adapters-bbeb2681/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.556246863Z namespaces/e2e-test-llm-with-lora-adapters-bbeb2681/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.556449868Z namespaces/e2e-test-llm-with-lora-adapters-bbeb2681/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.556615622Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.556626982Z namespaces/kserve-ci-e2e-test/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.556657643Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.556721144Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.556908608Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.557118173Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.557268177Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.557464841Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.557723407Z namespaces/kserve/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.557758478Z namespaces/kserve/kserve.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.55784818Z namespaces/kserve/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.557964653Z namespaces/kserve/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.558001594Z namespaces/kserve/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.558051405Z namespaces/kserve/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.558130087Z namespaces/kserve/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.558325911Z namespaces/kserve/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.558492735Z namespaces/kserve/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.558550966Z namespaces/kserve/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.558584217Z namespaces/kserve/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.558671619Z namespaces/kserve/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.55871262Z namespaces/kserve/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.558799462Z namespaces/kserve/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.558865904Z namespaces/kserve/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.558932695Z namespaces/kserve/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.559003767Z namespaces/kserve/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.559061088Z namespaces/kserve/coordination.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.559070129Z namespaces/kserve/coordination.k8s.io/leases/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.559078429Z namespaces/kserve/coordination.k8s.io/leases/kserve-controller-manager-leader-lock.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.559187331Z namespaces/kserve/coordination.k8s.io/leases/llminferenceservice-kserve-controller-manager.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.559265613Z namespaces/kserve/coordination.k8s.io/leases/odh-model-controller.opendatahub.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.559362845Z namespaces/kserve/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.559391586Z namespaces/kserve/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.559661342Z namespaces/kserve/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.559771815Z namespaces/kserve/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.560057332Z namespaces/kserve/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.560130733Z namespaces/kserve/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.560390699Z namespaces/kserve/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.560508452Z namespaces/kserve/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.560919431Z namespaces/kserve/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.561030394Z namespaces/kserve/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.561074355Z namespaces/kserve/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.561183847Z namespaces/kserve/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.561201708Z namespaces/kserve/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.56129103Z namespaces/kserve/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.561310481Z namespaces/kserve/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.561409853Z namespaces/kserve/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.561489135Z namespaces/kserve/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.561559106Z namespaces/kserve/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.561614568Z namespaces/kserve/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.561673439Z namespaces/kserve/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.56172432Z namespaces/kserve/monitoring.coreos.com/servicemonitors/model-serving-api-metrics.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.561819182Z namespaces/kserve/monitoring.coreos.com/servicemonitors/odh-model-controller-metrics-monitor.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.561923195Z namespaces/kserve/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.561950325Z namespaces/kserve/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.562025027Z namespaces/kserve/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.562032707Z namespaces/kserve/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.562059298Z namespaces/kserve/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.562221751Z namespaces/kserve/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.562483028Z namespaces/kserve/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.562623081Z namespaces/kserve/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.562829866Z namespaces/kserve/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.563050851Z namespaces/kserve/pods/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.563073151Z namespaces/kserve/pods/kserve-controller-manager-6ccfdc6bb8-tsbd7/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.563102412Z namespaces/kserve/pods/kserve-controller-manager-6ccfdc6bb8-tsbd7/kserve-controller-manager-6ccfdc6bb8-tsbd7.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.563195144Z namespaces/kserve/pods/kserve-controller-manager-6ccfdc6bb8-tsbd7/manager/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.563207354Z namespaces/kserve/pods/kserve-controller-manager-6ccfdc6bb8-tsbd7/manager/manager/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.563214554Z namespaces/kserve/pods/kserve-controller-manager-6ccfdc6bb8-tsbd7/manager/manager/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.563244385Z namespaces/kserve/pods/kserve-controller-manager-6ccfdc6bb8-tsbd7/manager/manager/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.563347008Z namespaces/kserve/pods/kserve-controller-manager-6ccfdc6bb8-tsbd7/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.56345205Z namespaces/kserve/pods/kserve-controller-manager-6ccfdc6bb8-tsbd7/manager/manager/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.56347032Z namespaces/kserve/pods/llmisvc-controller-manager-77cb7f7d68-fl4nw/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.563527332Z namespaces/kserve/pods/llmisvc-controller-manager-77cb7f7d68-fl4nw/llmisvc-controller-manager-77cb7f7d68-fl4nw.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.563620354Z namespaces/kserve/pods/llmisvc-controller-manager-77cb7f7d68-fl4nw/manager/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.563631604Z namespaces/kserve/pods/llmisvc-controller-manager-77cb7f7d68-fl4nw/manager/manager/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.563637054Z namespaces/kserve/pods/llmisvc-controller-manager-77cb7f7d68-fl4nw/manager/manager/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.563668375Z namespaces/kserve/pods/llmisvc-controller-manager-77cb7f7d68-fl4nw/manager/manager/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.578775255Z namespaces/kserve/pods/llmisvc-controller-manager-77cb7f7d68-fl4nw/manager/manager/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.578839096Z namespaces/kserve/pods/model-serving-api-7b954ffc95-gmv9t/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.578917298Z namespaces/kserve/pods/model-serving-api-7b954ffc95-gmv9t/model-serving-api-7b954ffc95-gmv9t.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.57899875Z namespaces/kserve/pods/model-serving-api-7b954ffc95-gmv9t/server/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.5790072Z namespaces/kserve/pods/model-serving-api-7b954ffc95-gmv9t/server/server/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.57901442Z namespaces/kserve/pods/model-serving-api-7b954ffc95-gmv9t/server/server/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.579084852Z namespaces/kserve/pods/model-serving-api-7b954ffc95-gmv9t/server/server/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.579167704Z namespaces/kserve/pods/model-serving-api-7b954ffc95-gmv9t/server/server/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.579238396Z namespaces/kserve/pods/model-serving-api-7b954ffc95-gmv9t/server/server/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.579284297Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-t5dhh/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.579319707Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-t5dhh/odh-model-controller-75d9cdbddf-t5dhh.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.57941278Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-t5dhh/manager/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.57942097Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-t5dhh/manager/manager/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.57943624Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-t5dhh/manager/manager/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.579475261Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-t5dhh/manager/manager/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.608701828Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-t5dhh/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.608744469Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-t5dhh/manager/manager/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.60882037Z namespaces/kserve/pods/s3-init-9m24t/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.608838951Z namespaces/kserve/pods/s3-init-9m24t/s3-init-9m24t.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.608978244Z namespaces/kserve/pods/s3-init-9m24t/download-hf-model/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.608993945Z namespaces/kserve/pods/s3-init-9m24t/download-hf-model/download-hf-model/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.608999575Z namespaces/kserve/pods/s3-init-9m24t/download-hf-model/download-hf-model/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.609012665Z namespaces/kserve/pods/s3-init-9m24t/download-hf-model/download-hf-model/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.609108287Z namespaces/kserve/pods/s3-init-9m24t/download-hf-model/download-hf-model/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.609173689Z namespaces/kserve/pods/s3-init-9m24t/download-hf-model/download-hf-model/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.60921889Z namespaces/kserve/pods/s3-init-9m24t/s3-init/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.60922714Z namespaces/kserve/pods/s3-init-9m24t/s3-init/s3-init/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.60923119Z namespaces/kserve/pods/s3-init-9m24t/s3-init/s3-init/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.609261271Z namespaces/kserve/pods/s3-init-9m24t/s3-init/s3-init/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.609542697Z namespaces/kserve/pods/s3-init-9m24t/s3-init/s3-init/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.609614879Z namespaces/kserve/pods/s3-init-9m24t/s3-init/s3-init/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.60965497Z namespaces/kserve/pods/seaweedfs-64568bcd49-bzpsq/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.609696301Z namespaces/kserve/pods/seaweedfs-64568bcd49-bzpsq/seaweedfs-64568bcd49-bzpsq.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.609770843Z namespaces/kserve/pods/seaweedfs-64568bcd49-bzpsq/seaweedfs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.609779523Z namespaces/kserve/pods/seaweedfs-64568bcd49-bzpsq/seaweedfs/seaweedfs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.609796693Z namespaces/kserve/pods/seaweedfs-64568bcd49-bzpsq/seaweedfs/seaweedfs/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.609845504Z namespaces/kserve/pods/seaweedfs-64568bcd49-bzpsq/seaweedfs/seaweedfs/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.610052639Z namespaces/kserve/pods/seaweedfs-64568bcd49-bzpsq/seaweedfs/seaweedfs/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.610121721Z namespaces/kserve/pods/seaweedfs-64568bcd49-bzpsq/seaweedfs/seaweedfs/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.610168802Z namespaces/kserve/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.610186572Z namespaces/kserve/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.610244163Z namespaces/kserve/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.610288304Z namespaces/kserve/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.610379617Z namespaces/kuadrant-system/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.610452438Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.61051896Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.610564861Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.610639723Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.610688074Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.610783646Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.611124984Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.61139542Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.611449421Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.611484692Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.611547824Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.611600495Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.611680067Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.611726158Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.611773489Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.611867251Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.611928753Z namespaces/kuadrant-system/coordination.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.611946683Z namespaces/kuadrant-system/coordination.k8s.io/leases/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.611998134Z namespaces/kuadrant-system/coordination.k8s.io/leases/3745a16e.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.612093566Z namespaces/kuadrant-system/coordination.k8s.io/leases/a3f98d6c.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.612186759Z namespaces/kuadrant-system/coordination.k8s.io/leases/aac3a15d.authorino.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.61227563Z namespaces/kuadrant-system/coordination.k8s.io/leases/f139389e.kuadrant.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.612321802Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.612372713Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.612553347Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.61268597Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.61313249Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.613210912Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.61353765Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.613671863Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.613918939Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.614042051Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.614061422Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.614184205Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.614210355Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.614271057Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.614307477Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.614389939Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.614468151Z namespaces/kuadrant-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.614515732Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.614556923Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.614608355Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.614626695Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.614722167Z namespaces/kuadrant-system/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.614736447Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.614779648Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.615032944Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.615247719Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.615404293Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.615619978Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.615833573Z namespaces/kuadrant-system/operators.coreos.com/installplans/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.615865064Z namespaces/kuadrant-system/operators.coreos.com/installplans/install-pfnxs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.616369395Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.616390086Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.616496348Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.61658023Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.616664442Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.616715413Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.616758354Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/kuadrant.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.616812206Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.616881387Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/authorino-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.616987379Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/dns-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.617138353Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/limitador-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.617233425Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/rhcl-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.617318737Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.617329038Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-s6nzs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.617375169Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-s6nzs/authorino-7cf4765b85-s6nzs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.61745977Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-s6nzs/authorino/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.617470271Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-s6nzs/authorino/authorino/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.617479361Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-s6nzs/authorino/authorino/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.617519892Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-s6nzs/authorino/authorino/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.627408801Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-s6nzs/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.627479313Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-s6nzs/authorino/authorino/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.627512453Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-n5pb8/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.627577385Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-n5pb8/authorino-operator-56fb6f6856-n5pb8.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.627692358Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-n5pb8/manager/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.627709198Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-n5pb8/manager/manager/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.627717148Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-n5pb8/manager/manager/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.627773329Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-n5pb8/manager/manager/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.627914193Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-n5pb8/manager/manager/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.628083696Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq5xn/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.628127768Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq5xn/dns-operator-controller-manager-ccbdf686-qq5xn.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.62823328Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq5xn/manager/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.62824005Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq5xn/manager/manager/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.62824635Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq5xn/manager/manager/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.628292131Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq5xn/manager/manager/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.628418194Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq5xn/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.628486236Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-qq5xn/manager/manager/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.62864006Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-c5j4w/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.62866321Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-c5j4w/kuadrant-console-plugin-598f86468b-c5j4w.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.628755412Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-c5j4w/kuadrant-console-plugin/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.628763852Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-c5j4w/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.628771683Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-c5j4w/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.628828474Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-c5j4w/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.629013968Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-c5j4w/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.62908573Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-c5j4w/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.629123991Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-fl2ps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.629169082Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-fl2ps/kuadrant-operator-controller-manager-66866f5fcc-fl2ps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.629298065Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-fl2ps/manager/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.629310545Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-fl2ps/manager/manager/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.629316065Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-fl2ps/manager/manager/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.629336985Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-fl2ps/manager/manager/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.63902941Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-fl2ps/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.639098402Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-fl2ps/manager/manager/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.639133812Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-cdzw4/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.639188104Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-cdzw4/limitador-limitador-85d9db4d5d-cdzw4.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.639283846Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-cdzw4/limitador/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.639299536Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-cdzw4/limitador/limitador/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.639305986Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-cdzw4/limitador/limitador/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.639341577Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-cdzw4/limitador/limitador/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.639412679Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-cdzw4/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.639493641Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-cdzw4/limitador/limitador/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.639551042Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-6fl6q/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.639619964Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-6fl6q/limitador-operator-controller-manager-774d94476f-6fl6q.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.639709006Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-6fl6q/manager/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.639719776Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-6fl6q/manager/manager/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.639725606Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-6fl6q/manager/manager/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.639766317Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-6fl6q/manager/manager/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.639953582Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-6fl6q/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.640024683Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-6fl6q/manager/manager/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.640068844Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.640107085Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.640162776Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.640253258Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.64031626Z namespaces/kube-node-lease/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.64032971Z namespaces/kube-node-lease/coordination.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.64033619Z namespaces/kube-node-lease/coordination.k8s.io/leases/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.640361821Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-132-194.ec2.internal.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.640455813Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-137-134.ec2.internal.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.640550155Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-143-86.ec2.internal.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.640584156Z namespaces/kube-node-lease/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.640615367Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.640673428Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.640913433Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.641146279Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.641288172Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.641520588Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.641817665Z namespaces/kube-public/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.641828235Z namespaces/kube-public/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.641832545Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.641897856Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.64206764Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.642249864Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.642377367Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.642708125Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.64292647Z namespaces/kube-system/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.642976171Z namespaces/kube-system/kube-system.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.643045303Z namespaces/kube-system/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.643084884Z namespaces/kube-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.643151165Z namespaces/kube-system/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.643193526Z namespaces/kube-system/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.643320569Z namespaces/kube-system/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.643399831Z namespaces/kube-system/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.643479943Z namespaces/kube-system/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.643545974Z namespaces/kube-system/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.643581235Z namespaces/kube-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.643650527Z namespaces/kube-system/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.643684748Z namespaces/kube-system/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.64377328Z namespaces/kube-system/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.643808661Z namespaces/kube-system/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.643846532Z namespaces/kube-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.643997235Z namespaces/kube-system/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.644020125Z namespaces/kube-system/coordination.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.644028126Z namespaces/kube-system/coordination.k8s.io/leases/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.644057937Z namespaces/kube-system/coordination.k8s.io/leases/apiserver-kpuals3x6x4o6unpkl6f6jxw64.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.644159319Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-cainjector-leader-election.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.644245881Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-controller.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.644333393Z namespaces/kube-system/coordination.k8s.io/leases/kube-controller-manager.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.644484256Z namespaces/kube-system/coordination.k8s.io/leases/kube-scheduler.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.644555938Z namespaces/kube-system/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.644601569Z namespaces/kube-system/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.644767493Z namespaces/kube-system/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.644895336Z namespaces/kube-system/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.645161712Z namespaces/kube-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.645235884Z namespaces/kube-system/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.64550673Z namespaces/kube-system/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.645629713Z namespaces/kube-system/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.64595977Z namespaces/kube-system/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.646016802Z namespaces/kube-system/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.646065253Z namespaces/kube-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.646116304Z namespaces/kube-system/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.646144805Z namespaces/kube-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.646223196Z namespaces/kube-system/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.646259337Z namespaces/kube-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.64635097Z namespaces/kube-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.646427641Z namespaces/kube-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.646489063Z namespaces/kube-system/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.646524454Z namespaces/kube-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.646577475Z namespaces/kube-system/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.646606256Z namespaces/kube-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.64680454Z namespaces/kube-system/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.64681295Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.646836131Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.647027895Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.647189439Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.647331822Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.647527977Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.647738672Z namespaces/kube-system/pods/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.647746072Z namespaces/kube-system/pods/global-pull-secret-syncer-88gv7/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.647797333Z namespaces/kube-system/pods/global-pull-secret-syncer-88gv7/global-pull-secret-syncer-88gv7.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.647912116Z namespaces/kube-system/pods/global-pull-secret-syncer-88gv7/global-pull-secret-syncer/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.647933576Z namespaces/kube-system/pods/global-pull-secret-syncer-88gv7/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.647941176Z namespaces/kube-system/pods/global-pull-secret-syncer-88gv7/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.647963667Z namespaces/kube-system/pods/global-pull-secret-syncer-88gv7/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.648281404Z namespaces/kube-system/pods/global-pull-secret-syncer-88gv7/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.648348616Z namespaces/kube-system/pods/global-pull-secret-syncer-88gv7/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.648415367Z namespaces/kube-system/pods/global-pull-secret-syncer-8jgtw/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.648472979Z namespaces/kube-system/pods/global-pull-secret-syncer-8jgtw/global-pull-secret-syncer-8jgtw.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.64853976Z namespaces/kube-system/pods/global-pull-secret-syncer-8jgtw/global-pull-secret-syncer/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.64855151Z namespaces/kube-system/pods/global-pull-secret-syncer-8jgtw/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.648557621Z namespaces/kube-system/pods/global-pull-secret-syncer-8jgtw/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.648591841Z namespaces/kube-system/pods/global-pull-secret-syncer-8jgtw/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.649037462Z namespaces/kube-system/pods/global-pull-secret-syncer-8jgtw/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.649066922Z namespaces/kube-system/pods/global-pull-secret-syncer-8jgtw/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.649104283Z namespaces/kube-system/pods/global-pull-secret-syncer-g6775/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.649152024Z namespaces/kube-system/pods/global-pull-secret-syncer-g6775/global-pull-secret-syncer-g6775.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.649298018Z namespaces/kube-system/pods/global-pull-secret-syncer-g6775/global-pull-secret-syncer/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.649308478Z namespaces/kube-system/pods/global-pull-secret-syncer-g6775/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.649312828Z namespaces/kube-system/pods/global-pull-secret-syncer-g6775/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.649316658Z namespaces/kube-system/pods/global-pull-secret-syncer-g6775/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.649604455Z namespaces/kube-system/pods/global-pull-secret-syncer-g6775/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.649680737Z namespaces/kube-system/pods/global-pull-secret-syncer-g6775/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.649692717Z namespaces/kube-system/pods/konnectivity-agent-gdxlg/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.649745158Z namespaces/kube-system/pods/konnectivity-agent-gdxlg/konnectivity-agent-gdxlg.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.64982757Z namespaces/kube-system/pods/konnectivity-agent-gdxlg/konnectivity-agent/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.64983462Z namespaces/kube-system/pods/konnectivity-agent-gdxlg/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.649847011Z namespaces/kube-system/pods/konnectivity-agent-gdxlg/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.649941393Z namespaces/kube-system/pods/konnectivity-agent-gdxlg/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.651263593Z namespaces/kube-system/pods/konnectivity-agent-gdxlg/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.651332485Z namespaces/kube-system/pods/konnectivity-agent-gdxlg/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.651388466Z namespaces/kube-system/pods/konnectivity-agent-vclxn/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.651455938Z namespaces/kube-system/pods/konnectivity-agent-vclxn/konnectivity-agent-vclxn.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.651572491Z namespaces/kube-system/pods/konnectivity-agent-vclxn/konnectivity-agent/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.651580591Z namespaces/kube-system/pods/konnectivity-agent-vclxn/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.651584941Z namespaces/kube-system/pods/konnectivity-agent-vclxn/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.651608751Z namespaces/kube-system/pods/konnectivity-agent-vclxn/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.652554873Z namespaces/kube-system/pods/konnectivity-agent-vclxn/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.652628935Z namespaces/kube-system/pods/konnectivity-agent-vclxn/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.652688276Z namespaces/kube-system/pods/konnectivity-agent-zxf8h/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.652732517Z namespaces/kube-system/pods/konnectivity-agent-zxf8h/konnectivity-agent-zxf8h.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.652812789Z namespaces/kube-system/pods/konnectivity-agent-zxf8h/konnectivity-agent/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.652821789Z namespaces/kube-system/pods/konnectivity-agent-zxf8h/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.65282678Z namespaces/kube-system/pods/konnectivity-agent-zxf8h/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.65285051Z namespaces/kube-system/pods/konnectivity-agent-zxf8h/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.653802652Z namespaces/kube-system/pods/konnectivity-agent-zxf8h/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.653873804Z namespaces/kube-system/pods/konnectivity-agent-zxf8h/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.653932155Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-194.ec2.internal/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.653973486Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-194.ec2.internal/kube-apiserver-proxy-ip-10-0-132-194.ec2.internal.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.654066058Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-194.ec2.internal/haproxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.654074208Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-194.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.654078229Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-194.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.654114569Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-194.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.654356165Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-194.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.654409396Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-194.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.654853236Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-134.ec2.internal/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.654904017Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-134.ec2.internal/kube-apiserver-proxy-ip-10-0-137-134.ec2.internal.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.655037591Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-134.ec2.internal/haproxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.655052821Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-134.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.655058051Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-134.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.655140723Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-134.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.655555853Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-134.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.655571353Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-134.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.655705926Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-143-86.ec2.internal/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.655755607Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-143-86.ec2.internal/kube-apiserver-proxy-ip-10-0-143-86.ec2.internal.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.655833499Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-143-86.ec2.internal/haproxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.655842699Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-143-86.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.655847559Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-143-86.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.655910541Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-143-86.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.65629792Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-143-86.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.656414303Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-143-86.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.656425663Z namespaces/kube-system/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.656484814Z namespaces/kube-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.656536915Z namespaces/kube-system/rbac.authorization.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.656570716Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.656579336Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-authentication-reader.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.656671318Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.65672613Z namespaces/kube-system/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.656773121Z namespaces/kube-system/rbac.authorization.k8s.io/roles/extension-apiserver-authentication-reader.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.656826822Z namespaces/kube-system/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.656879183Z namespaces/kube-system/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.657035907Z namespaces/observability/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.657048127Z namespaces/observability/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.657054517Z namespaces/observability/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.657063658Z namespaces/observability/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.657268382Z namespaces/observability/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.657470727Z namespaces/observability/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.65760447Z namespaces/observability/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.657814495Z namespaces/observability/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.65805017Z namespaces/open-cluster-management-764fc977-3a16-4fab-8f50-575dca4dd/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.658063491Z namespaces/open-cluster-management-764fc977-3a16-4fab-8f50-575dca4dd/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.658074501Z namespaces/open-cluster-management-764fc977-3a16-4fab-8f50-575dca4dd/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.658139052Z namespaces/open-cluster-management-764fc977-3a16-4fab-8f50-575dca4dd/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.658292256Z namespaces/open-cluster-management-764fc977-3a16-4fab-8f50-575dca4dd/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.658510511Z namespaces/open-cluster-management-764fc977-3a16-4fab-8f50-575dca4dd/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.658828058Z namespaces/open-cluster-management-764fc977-3a16-4fab-8f50-575dca4dd/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.659045584Z namespaces/open-cluster-management-764fc977-3a16-4fab-8f50-575dca4dd/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.659248638Z namespaces/open-cluster-management-agent-addon/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.659259448Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.659265669Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.659278409Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/cluster-proxy.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.659386441Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/work-manager.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.659418302Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.659431242Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.659489824Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.659647727Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.659828042Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.659992475Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.66019047Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.660410905Z namespaces/openshift-apiserver-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.660458876Z namespaces/openshift-apiserver-operator/openshift-apiserver-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.660529748Z namespaces/openshift-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.660580989Z namespaces/openshift-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.660649361Z namespaces/openshift-apiserver-operator/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.660698672Z namespaces/openshift-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.660789624Z namespaces/openshift-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.660867616Z namespaces/openshift-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.660971888Z namespaces/openshift-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.661023209Z namespaces/openshift-apiserver-operator/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.66106754Z namespaces/openshift-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.661112141Z namespaces/openshift-apiserver-operator/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.661169773Z namespaces/openshift-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.661252095Z namespaces/openshift-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.661296576Z namespaces/openshift-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.661334466Z namespaces/openshift-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.661421779Z namespaces/openshift-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.66147609Z namespaces/openshift-apiserver-operator/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.661526571Z namespaces/openshift-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.662588026Z namespaces/openshift-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.662698988Z namespaces/openshift-apiserver-operator/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.6627888Z namespaces/openshift-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.662922123Z namespaces/openshift-apiserver-operator/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.663008775Z namespaces/openshift-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.663125708Z namespaces/openshift-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.663311432Z namespaces/openshift-apiserver-operator/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.663821924Z namespaces/openshift-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.663843795Z namespaces/openshift-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.663953727Z namespaces/openshift-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.663985128Z namespaces/openshift-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.66406077Z namespaces/openshift-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.66408514Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.664188752Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.664266324Z namespaces/openshift-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.664323806Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.664368367Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.664433528Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.664486239Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/openshift-apiserver-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.664533331Z namespaces/openshift-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.664829517Z namespaces/openshift-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.664835927Z namespaces/openshift-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.664839978Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.664849888Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.664877848Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.665068523Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.665206096Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.665414321Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.665623406Z namespaces/openshift-apiserver-operator/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.665668997Z namespaces/openshift-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.665709788Z namespaces/openshift-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.665759299Z namespaces/openshift-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.665878012Z namespaces/openshift-apiserver/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.665939103Z namespaces/openshift-apiserver/openshift-apiserver.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.666022545Z namespaces/openshift-apiserver/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.666055946Z namespaces/openshift-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.666113987Z namespaces/openshift-apiserver/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.666170869Z namespaces/openshift-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.66626124Z namespaces/openshift-apiserver/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.666340782Z namespaces/openshift-apiserver/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.666418974Z namespaces/openshift-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.666461505Z namespaces/openshift-apiserver/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.666588948Z namespaces/openshift-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.66665589Z namespaces/openshift-apiserver/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.666690321Z namespaces/openshift-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.666790703Z namespaces/openshift-apiserver/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.666831864Z namespaces/openshift-apiserver/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.666876075Z namespaces/openshift-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.667044249Z namespaces/openshift-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.66711045Z namespaces/openshift-apiserver/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.667137211Z namespaces/openshift-apiserver/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.667264574Z namespaces/openshift-apiserver/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.667321865Z namespaces/openshift-apiserver/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.667401767Z namespaces/openshift-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.667490829Z namespaces/openshift-apiserver/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.667566181Z namespaces/openshift-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.667692954Z namespaces/openshift-apiserver/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.667847957Z namespaces/openshift-apiserver/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.667908979Z namespaces/openshift-apiserver/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.66796482Z namespaces/openshift-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.668010891Z namespaces/openshift-apiserver/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.668071762Z namespaces/openshift-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.668132644Z namespaces/openshift-apiserver/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.668159464Z namespaces/openshift-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.668300178Z namespaces/openshift-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.668367959Z namespaces/openshift-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.66842315Z namespaces/openshift-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.668455261Z namespaces/openshift-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.668516883Z namespaces/openshift-apiserver/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.668567124Z namespaces/openshift-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.668610035Z namespaces/openshift-apiserver/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.668631195Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.668691447Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.668884031Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.669063375Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.669210219Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.669384553Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.669562497Z namespaces/openshift-apiserver/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.669585808Z namespaces/openshift-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.669658769Z namespaces/openshift-apiserver/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.66967329Z namespaces/openshift-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.669734811Z namespaces/openshift-authentication-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.669746531Z namespaces/openshift-authentication-operator/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.669756792Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.669808883Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/authentication-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.669917145Z namespaces/openshift-authentication-operator/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.669944076Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.670002827Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.670161801Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.670337455Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.670485298Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.670719874Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.671214805Z namespaces/openshift-authentication/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.671235816Z namespaces/openshift-authentication/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.671247266Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.671284817Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.671461281Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.671635805Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.671781918Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.672010114Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.672229429Z namespaces/openshift-cloud-controller-manager/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.672243759Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.672250179Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.67228059Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/cloud-controller-manager.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.672363682Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.672376352Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.672421793Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.672590157Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.672766441Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.672930755Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.673116139Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.673332654Z namespaces/openshift-cloud-credential-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.673343415Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.673353675Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.673385636Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/cloud-credential-operator-leader.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.673444877Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.673455307Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.673493208Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/cloud-credential-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.6735593Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.67356977Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.673620801Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.673771194Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.674054521Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.674186244Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.674375408Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.67488919Z namespaces/openshift-cloud-network-config-controller/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.674933271Z namespaces/openshift-cloud-network-config-controller/openshift-cloud-network-config-controller.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.675003773Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.675050754Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.675111885Z namespaces/openshift-cloud-network-config-controller/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.675158437Z namespaces/openshift-cloud-network-config-controller/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.675241968Z namespaces/openshift-cloud-network-config-controller/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.67532112Z namespaces/openshift-cloud-network-config-controller/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.675399642Z namespaces/openshift-cloud-network-config-controller/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.675439133Z namespaces/openshift-cloud-network-config-controller/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.675483694Z namespaces/openshift-cloud-network-config-controller/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.675527035Z namespaces/openshift-cloud-network-config-controller/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.675589196Z namespaces/openshift-cloud-network-config-controller/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.675699119Z namespaces/openshift-cloud-network-config-controller/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.67576404Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.675821332Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.675965475Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.675994616Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.676000626Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.676070448Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/cloud-network-config-controller-lock.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.67616726Z namespaces/openshift-cloud-network-config-controller/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.676214751Z namespaces/openshift-cloud-network-config-controller/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.676345534Z namespaces/openshift-cloud-network-config-controller/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.676434946Z namespaces/openshift-cloud-network-config-controller/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.676515148Z namespaces/openshift-cloud-network-config-controller/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.67661375Z namespaces/openshift-cloud-network-config-controller/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.676718533Z namespaces/openshift-cloud-network-config-controller/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.676890187Z namespaces/openshift-cloud-network-config-controller/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.67703135Z namespaces/openshift-cloud-network-config-controller/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.677938031Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.677992252Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/cloud-network-config-controller.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.678058284Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.678091465Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.678139575Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.678180436Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.678263618Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.678292329Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.678372781Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.678459763Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.678493874Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.678532325Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.678583446Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.678635297Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.678693818Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.678707819Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.6787457Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.678906613Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.679095638Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.679218471Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.679443256Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.67962152Z namespaces/openshift-cloud-network-config-controller/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.679662771Z namespaces/openshift-cloud-network-config-controller/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.679710372Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.679724652Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.679759763Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/cloud-network-config-controller-rb.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.679813034Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.679883586Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/cloud-network-config-controller.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.679924297Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.679967628Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.68005732Z namespaces/openshift-cluster-csi-drivers/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.680170653Z namespaces/openshift-cluster-csi-drivers/openshift-cluster-csi-drivers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.680251424Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.680289725Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.680374497Z namespaces/openshift-cluster-csi-drivers/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.680429088Z namespaces/openshift-cluster-csi-drivers/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.680574012Z namespaces/openshift-cluster-csi-drivers/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.680670404Z namespaces/openshift-cluster-csi-drivers/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.680761196Z namespaces/openshift-cluster-csi-drivers/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.680815837Z namespaces/openshift-cluster-csi-drivers/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.680884909Z namespaces/openshift-cluster-csi-drivers/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.68094812Z namespaces/openshift-cluster-csi-drivers/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.681000652Z namespaces/openshift-cluster-csi-drivers/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.681096884Z namespaces/openshift-cluster-csi-drivers/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.681168446Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.681247987Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.68135295Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.681425572Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.681441432Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.681467783Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ebs-csi-aws-com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.681605166Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-attacher-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.681701438Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-resizer-ebs-csi-aws-com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.681813031Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-snapshotter-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.682083827Z namespaces/openshift-cluster-csi-drivers/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.682142948Z namespaces/openshift-cluster-csi-drivers/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.682257341Z namespaces/openshift-cluster-csi-drivers/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.682374374Z namespaces/openshift-cluster-csi-drivers/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.682738492Z namespaces/openshift-cluster-csi-drivers/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.682830404Z namespaces/openshift-cluster-csi-drivers/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.683335076Z namespaces/openshift-cluster-csi-drivers/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.68350956Z namespaces/openshift-cluster-csi-drivers/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.683641623Z namespaces/openshift-cluster-csi-drivers/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.683698314Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.683762266Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/aws-ebs-csi-driver-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.683812357Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.683896919Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.68394752Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.684010662Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.684049363Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.684113424Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.684209946Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.684297908Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.684347349Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.684404971Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.684450382Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.684509183Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.684602765Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.684625766Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.684672367Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.684849521Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.685062206Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.68521765Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.685427454Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.685645929Z namespaces/openshift-cluster-csi-drivers/pods/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.68565914Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8qwz2/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.685703211Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8qwz2/aws-ebs-csi-driver-node-8qwz2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.685834724Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8qwz2/csi-driver/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.685847254Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8qwz2/csi-driver/csi-driver/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.685853584Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8qwz2/csi-driver/csi-driver/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.685961127Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8qwz2/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.686025548Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8qwz2/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.68610885Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8qwz2/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.686158771Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8qwz2/csi-liveness-probe/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.686170221Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8qwz2/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.686178632Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8qwz2/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.686232863Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8qwz2/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.686314645Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8qwz2/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.686407977Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8qwz2/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.686457898Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8qwz2/csi-node-driver-registrar/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.686468988Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8qwz2/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.686473319Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8qwz2/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.6865281Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8qwz2/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.686658053Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8qwz2/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.686734574Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-8qwz2/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.686795946Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-spqqt/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.686848887Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-spqqt/aws-ebs-csi-driver-node-spqqt.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.68697898Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-spqqt/csi-driver/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.686995961Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-spqqt/csi-driver/csi-driver/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.687008961Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-spqqt/csi-driver/csi-driver/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.687053282Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-spqqt/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.687151314Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-spqqt/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.687242106Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-spqqt/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.687285077Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-spqqt/csi-liveness-probe/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.687293187Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-spqqt/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.687305558Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-spqqt/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.687354169Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-spqqt/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.687454531Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-spqqt/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.687529613Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-spqqt/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.687572954Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-spqqt/csi-node-driver-registrar/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.687581684Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-spqqt/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.687592445Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-spqqt/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.687652806Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-spqqt/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.687756238Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-spqqt/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.68782615Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-spqqt/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.687978063Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wp9wk/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.688020784Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wp9wk/aws-ebs-csi-driver-node-wp9wk.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.688193338Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wp9wk/csi-driver/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.688398623Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wp9wk/csi-driver/csi-driver/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.688414833Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wp9wk/csi-driver/csi-driver/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.688463215Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wp9wk/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.688583937Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wp9wk/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.688670839Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wp9wk/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.688740011Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wp9wk/csi-liveness-probe/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.688787042Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wp9wk/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.688798572Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wp9wk/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.688884534Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wp9wk/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.689013427Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wp9wk/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.689091139Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wp9wk/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.689154121Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wp9wk/csi-node-driver-registrar/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.689181421Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wp9wk/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.689193322Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wp9wk/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.689221782Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wp9wk/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.689319565Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wp9wk/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.689406416Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wp9wk/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.689486838Z namespaces/openshift-cluster-csi-drivers/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.689512769Z namespaces/openshift-cluster-csi-drivers/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.68958458Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.689601631Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.689649432Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/aws-ebs-csi-driver-operator-rolebinding.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.689726954Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.689779665Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/aws-ebs-csi-driver-operator-role.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.689876277Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.69000111Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.690097503Z namespaces/openshift-cluster-machine-approver/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.690573343Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.690597404Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.690662976Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/cluster-machine-approver-leader.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.690741727Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.690754088Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.690802429Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/cluster-machine-approver.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.690907321Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.690945982Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.691015584Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.691418513Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.691627978Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.691882314Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.692115779Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.692335964Z namespaces/openshift-cluster-node-tuning-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.692407796Z namespaces/openshift-cluster-node-tuning-operator/openshift-cluster-node-tuning-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.692506818Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.69258825Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.692662842Z namespaces/openshift-cluster-node-tuning-operator/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.692699673Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.692852436Z namespaces/openshift-cluster-node-tuning-operator/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.692960309Z namespaces/openshift-cluster-node-tuning-operator/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.69303952Z namespaces/openshift-cluster-node-tuning-operator/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.693078641Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.693124453Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/tuned.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.693204374Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.693242085Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.693288086Z namespaces/openshift-cluster-node-tuning-operator/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.693329177Z namespaces/openshift-cluster-node-tuning-operator/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.693417849Z namespaces/openshift-cluster-node-tuning-operator/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.69347045Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.693523982Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.693607814Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.693662505Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.693677605Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.693736867Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/node-tuning-operator-lock.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.693832709Z namespaces/openshift-cluster-node-tuning-operator/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.693898851Z namespaces/openshift-cluster-node-tuning-operator/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.694852242Z namespaces/openshift-cluster-node-tuning-operator/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.694970055Z namespaces/openshift-cluster-node-tuning-operator/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.695122629Z namespaces/openshift-cluster-node-tuning-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.69518725Z namespaces/openshift-cluster-node-tuning-operator/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.695391415Z namespaces/openshift-cluster-node-tuning-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.695550239Z namespaces/openshift-cluster-node-tuning-operator/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.695677222Z namespaces/openshift-cluster-node-tuning-operator/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.695744533Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.695806025Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/cluster-node-tuning-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.695883466Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.695915057Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.695980269Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.69602203Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.696077621Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.696119472Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.696231225Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.696296616Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.696370718Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.696433789Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.696509231Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.696561292Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/node-tuning-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.696651384Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.696690605Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/node-tuning-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.696754446Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.696798927Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.69690249Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.696925901Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.696972912Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.697193207Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.697411312Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.697563495Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.697845512Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.698134958Z namespaces/openshift-cluster-node-tuning-operator/pods/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.698163869Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-4wmc2/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.69821323Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-4wmc2/tuned-4wmc2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.698341533Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-4wmc2/tuned/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.698363584Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-4wmc2/tuned/tuned/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.698369994Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-4wmc2/tuned/tuned/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.698441206Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-4wmc2/tuned/tuned/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.698585949Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-4wmc2/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.698688091Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-4wmc2/tuned/tuned/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.698754593Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jv9gw/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.698812354Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jv9gw/tuned-jv9gw.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.698942327Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jv9gw/tuned/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.698960568Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jv9gw/tuned/tuned/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.698967078Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jv9gw/tuned/tuned/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.698995478Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jv9gw/tuned/tuned/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.699204123Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jv9gw/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.699306866Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jv9gw/tuned/tuned/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.699372417Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-vh2nd/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.699451549Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-vh2nd/tuned-vh2nd.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.699558062Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-vh2nd/tuned/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.699566742Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-vh2nd/tuned/tuned/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.699570832Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-vh2nd/tuned/tuned/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.699626383Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-vh2nd/tuned/tuned/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.699795887Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-vh2nd/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.69992388Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-vh2nd/tuned/tuned/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.699973471Z namespaces/openshift-cluster-node-tuning-operator/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.700035662Z namespaces/openshift-cluster-node-tuning-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.700117584Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.700174056Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.700251878Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.700277238Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.70035615Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-132-194.ec2.internal.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.700516284Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-137-134.ec2.internal.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.700610026Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-143-86.ec2.internal.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.700693398Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.700741219Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/default.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.700845131Z namespaces/openshift-cluster-samples-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.700938703Z namespaces/openshift-cluster-samples-operator/openshift-cluster-samples-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.701028216Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.701084757Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.701201529Z namespaces/openshift-cluster-samples-operator/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.70122054Z namespaces/openshift-cluster-samples-operator/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.701327623Z namespaces/openshift-cluster-samples-operator/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.701460875Z namespaces/openshift-cluster-samples-operator/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.70163517Z namespaces/openshift-cluster-samples-operator/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.701676031Z namespaces/openshift-cluster-samples-operator/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.701737992Z namespaces/openshift-cluster-samples-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.701805614Z namespaces/openshift-cluster-samples-operator/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.701873835Z namespaces/openshift-cluster-samples-operator/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.701999118Z namespaces/openshift-cluster-samples-operator/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.702066029Z namespaces/openshift-cluster-samples-operator/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.702122801Z namespaces/openshift-cluster-samples-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.702214923Z namespaces/openshift-cluster-samples-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.702334566Z namespaces/openshift-cluster-samples-operator/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.702371957Z namespaces/openshift-cluster-samples-operator/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.702607542Z namespaces/openshift-cluster-samples-operator/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.702737945Z namespaces/openshift-cluster-samples-operator/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.702923779Z namespaces/openshift-cluster-samples-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.703213646Z namespaces/openshift-cluster-samples-operator/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.703245777Z namespaces/openshift-cluster-samples-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.70335889Z namespaces/openshift-cluster-samples-operator/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.703560744Z namespaces/openshift-cluster-samples-operator/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.703625336Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.703664327Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.703762999Z namespaces/openshift-cluster-samples-operator/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.70382592Z namespaces/openshift-cluster-samples-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.703945393Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.703993004Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.704102947Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.704202409Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.70425292Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.704306232Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.704371443Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.704415744Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/samples-operator-alerts.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.704502146Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.704555807Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/cluster-samples-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.704623609Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.7046602Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.704731481Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.704741631Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.704778592Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.704998457Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.705176672Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.705322005Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.70553377Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.705733994Z namespaces/openshift-cluster-samples-operator/pods/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.705743385Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-ftc4g/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.705790056Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-ftc4g/cluster-samples-operator-6748489656-ftc4g.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.705921979Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-ftc4g/cluster-samples-operator-watch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.705937049Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-ftc4g/cluster-samples-operator-watch/cluster-samples-operator-watch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.705941819Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-ftc4g/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.70597785Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-ftc4g/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.706095263Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-ftc4g/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.706178025Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-ftc4g/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.706239696Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-ftc4g/cluster-samples-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.706250146Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-ftc4g/cluster-samples-operator/cluster-samples-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.706256377Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-ftc4g/cluster-samples-operator/cluster-samples-operator/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.706319628Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-ftc4g/cluster-samples-operator/cluster-samples-operator/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.707029304Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-ftc4g/cluster-samples-operator/cluster-samples-operator/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.707144997Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-ftc4g/cluster-samples-operator/cluster-samples-operator/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.707219999Z namespaces/openshift-cluster-samples-operator/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.70726434Z namespaces/openshift-cluster-samples-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.707346502Z namespaces/openshift-cluster-samples-operator/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.707390783Z namespaces/openshift-cluster-samples-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.707503986Z namespaces/openshift-cluster-storage-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.707564587Z namespaces/openshift-cluster-storage-operator/openshift-cluster-storage-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.707642229Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.70768808Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.707766152Z namespaces/openshift-cluster-storage-operator/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.707830173Z namespaces/openshift-cluster-storage-operator/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.707969746Z namespaces/openshift-cluster-storage-operator/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.708096769Z namespaces/openshift-cluster-storage-operator/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.708248453Z namespaces/openshift-cluster-storage-operator/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.708307234Z namespaces/openshift-cluster-storage-operator/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.708347385Z namespaces/openshift-cluster-storage-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.708405066Z namespaces/openshift-cluster-storage-operator/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.708444757Z namespaces/openshift-cluster-storage-operator/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.708530469Z namespaces/openshift-cluster-storage-operator/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.708587471Z namespaces/openshift-cluster-storage-operator/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.708621001Z namespaces/openshift-cluster-storage-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.708710223Z namespaces/openshift-cluster-storage-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.708751244Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.708768855Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.708809656Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/data-source-validator-leader.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.708937909Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/snapshot-controller-leader.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.70901151Z namespaces/openshift-cluster-storage-operator/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.709048421Z namespaces/openshift-cluster-storage-operator/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.709178504Z namespaces/openshift-cluster-storage-operator/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.709303447Z namespaces/openshift-cluster-storage-operator/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.709817019Z namespaces/openshift-cluster-storage-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.709935092Z namespaces/openshift-cluster-storage-operator/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.710055534Z namespaces/openshift-cluster-storage-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.710196288Z namespaces/openshift-cluster-storage-operator/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.710370922Z namespaces/openshift-cluster-storage-operator/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.710404823Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.710454794Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/csi-snapshot-controller-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.710557656Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.710589417Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.710686439Z namespaces/openshift-cluster-storage-operator/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.710756321Z namespaces/openshift-cluster-storage-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.710840293Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.710911474Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.711014117Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.711113389Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.711183261Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.711250172Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.711322134Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.711377155Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/prometheus.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.711464297Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.711509318Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/cluster-storage-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.71157609Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.711630531Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.711719983Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.711730613Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.711813535Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.711994479Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.712171343Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.712316127Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.712544272Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.712730316Z namespaces/openshift-cluster-storage-operator/pods/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.712739367Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-q8p9z/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.712790488Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-q8p9z/volume-data-source-validator-549c6d54cf-q8p9z.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.71288185Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-q8p9z/volume-data-source-validator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.71290194Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-q8p9z/volume-data-source-validator/volume-data-source-validator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.712934811Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-q8p9z/volume-data-source-validator/volume-data-source-validator/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.712950561Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-q8p9z/volume-data-source-validator/volume-data-source-validator/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.713083705Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-q8p9z/volume-data-source-validator/volume-data-source-validator/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.713165977Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-q8p9z/volume-data-source-validator/volume-data-source-validator/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.713215378Z namespaces/openshift-cluster-storage-operator/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.713269209Z namespaces/openshift-cluster-storage-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.7133306Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.713339601Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.713393622Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.713503634Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.713576686Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.713675058Z namespaces/openshift-cluster-storage-operator/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.71375368Z namespaces/openshift-cluster-storage-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.713892654Z namespaces/openshift-cluster-version/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.714117649Z namespaces/openshift-cluster-version/openshift-cluster-version.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.714222371Z namespaces/openshift-cluster-version/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.714280532Z namespaces/openshift-cluster-version/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.714313483Z namespaces/openshift-cluster-version/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.714379415Z namespaces/openshift-cluster-version/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.71462269Z namespaces/openshift-cluster-version/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.714747793Z namespaces/openshift-cluster-version/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.714885466Z namespaces/openshift-cluster-version/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.714960048Z namespaces/openshift-cluster-version/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.715006009Z namespaces/openshift-cluster-version/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.715105102Z namespaces/openshift-cluster-version/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.715155663Z namespaces/openshift-cluster-version/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.715282115Z namespaces/openshift-cluster-version/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.715364408Z namespaces/openshift-cluster-version/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.715508021Z namespaces/openshift-cluster-version/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.715614553Z namespaces/openshift-cluster-version/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.715700015Z namespaces/openshift-cluster-version/coordination.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.715709525Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.715762967Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/version.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.71590376Z namespaces/openshift-cluster-version/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.715954471Z namespaces/openshift-cluster-version/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.716090024Z namespaces/openshift-cluster-version/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.716205777Z namespaces/openshift-cluster-version/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.71634016Z namespaces/openshift-cluster-version/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.716603546Z namespaces/openshift-cluster-version/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.716716429Z namespaces/openshift-cluster-version/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.716903133Z namespaces/openshift-cluster-version/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.717145869Z namespaces/openshift-cluster-version/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.717240811Z namespaces/openshift-cluster-version/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.717306602Z namespaces/openshift-cluster-version/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.717396564Z namespaces/openshift-cluster-version/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.717425175Z namespaces/openshift-cluster-version/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.717513907Z namespaces/openshift-cluster-version/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.717558948Z namespaces/openshift-cluster-version/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.717660181Z namespaces/openshift-cluster-version/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.717778104Z namespaces/openshift-cluster-version/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.717831015Z namespaces/openshift-cluster-version/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.717916807Z namespaces/openshift-cluster-version/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.717984908Z namespaces/openshift-cluster-version/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.71804815Z namespaces/openshift-cluster-version/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.718264735Z namespaces/openshift-cluster-version/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.718276435Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.718350977Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.718603412Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.718832388Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.719028792Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.719295868Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.719530904Z namespaces/openshift-cluster-version/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.719639376Z namespaces/openshift-cluster-version/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.719720478Z namespaces/openshift-cluster-version/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.71977294Z namespaces/openshift-cluster-version/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.720071286Z namespaces/openshift-config-managed/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.720126008Z namespaces/openshift-config-managed/openshift-config-managed.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.720360323Z namespaces/openshift-config-managed/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.720420974Z namespaces/openshift-config-managed/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.720518807Z namespaces/openshift-config-managed/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.720547957Z namespaces/openshift-config-managed/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.720803953Z namespaces/openshift-config-managed/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.720935386Z namespaces/openshift-config-managed/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.721036159Z namespaces/openshift-config-managed/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.72109695Z namespaces/openshift-config-managed/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.721161712Z namespaces/openshift-config-managed/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.721224213Z namespaces/openshift-config-managed/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.721276474Z namespaces/openshift-config-managed/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.721392957Z namespaces/openshift-config-managed/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.721475729Z namespaces/openshift-config-managed/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.72153054Z namespaces/openshift-config-managed/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.721644433Z namespaces/openshift-config-managed/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.721751676Z namespaces/openshift-config-managed/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.721806267Z namespaces/openshift-config-managed/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.725275217Z namespaces/openshift-config-managed/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.725346829Z namespaces/openshift-config-managed/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.725486622Z namespaces/openshift-config-managed/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.725573064Z namespaces/openshift-config-managed/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.725658026Z namespaces/openshift-config-managed/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.725807859Z namespaces/openshift-config-managed/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.726010394Z namespaces/openshift-config-managed/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.726084216Z namespaces/openshift-config-managed/core/configmaps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.726134557Z namespaces/openshift-config-managed/core/configmaps/console-public.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.7262594Z namespaces/openshift-config-managed/core/configmaps/openshift-network-features.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.726317501Z namespaces/openshift-config-managed/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.726393873Z namespaces/openshift-config-managed/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.726429934Z namespaces/openshift-config-managed/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.726492695Z namespaces/openshift-config-managed/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.726580537Z namespaces/openshift-config-managed/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.726643839Z namespaces/openshift-config-managed/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.726777942Z namespaces/openshift-config-managed/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.726889064Z namespaces/openshift-config-managed/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.726947866Z namespaces/openshift-config-managed/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.727004257Z namespaces/openshift-config-managed/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.727092809Z namespaces/openshift-config-managed/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.72715911Z namespaces/openshift-config-managed/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.727242323Z namespaces/openshift-config-managed/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.727252553Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.727330354Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.727594751Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.728121023Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.728278587Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.728538053Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.728831799Z namespaces/openshift-config-managed/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.728911151Z namespaces/openshift-config-managed/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.728980623Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.728990313Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.729059145Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/openshift-network-public-role-binding.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.729139756Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.729195008Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/openshift-network-public-role.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.72927525Z namespaces/openshift-config-managed/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.729333151Z namespaces/openshift-config-managed/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.729402813Z namespaces/openshift-config-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.729411013Z namespaces/openshift-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.729421173Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.729483555Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/config-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.729592337Z namespaces/openshift-config-operator/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.729617418Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.729664919Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.729875844Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.730086628Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.730220412Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.730419886Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.730771234Z namespaces/openshift-config/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.730814915Z namespaces/openshift-config/openshift-config.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.730916917Z namespaces/openshift-config/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.730952168Z namespaces/openshift-config/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.73101676Z namespaces/openshift-config/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.731059691Z namespaces/openshift-config/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.731146193Z namespaces/openshift-config/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.731228535Z namespaces/openshift-config/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.731302397Z namespaces/openshift-config/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.731358358Z namespaces/openshift-config/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.731401059Z namespaces/openshift-config/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.73146067Z namespaces/openshift-config/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.731510631Z namespaces/openshift-config/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.731589383Z namespaces/openshift-config/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.731634274Z namespaces/openshift-config/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.731670845Z namespaces/openshift-config/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.731764007Z namespaces/openshift-config/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.731801788Z namespaces/openshift-config/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.731846979Z namespaces/openshift-config/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.731998783Z namespaces/openshift-config/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.732073444Z namespaces/openshift-config/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.732158216Z namespaces/openshift-config/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.732240828Z namespaces/openshift-config/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.73232259Z namespaces/openshift-config/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.732481194Z namespaces/openshift-config/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.732635267Z namespaces/openshift-config/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.732676638Z namespaces/openshift-config/core/secrets/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.73274594Z namespaces/openshift-config/core/secrets/pull-secret.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.73276394Z namespaces/openshift-config/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.732814621Z namespaces/openshift-config/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.732901763Z namespaces/openshift-config/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.732947855Z namespaces/openshift-config/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.733007576Z namespaces/openshift-config/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.733056307Z namespaces/openshift-config/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.733152459Z namespaces/openshift-config/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.733207471Z namespaces/openshift-config/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.733266602Z namespaces/openshift-config/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.733308633Z namespaces/openshift-config/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.733365194Z namespaces/openshift-config/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.733407145Z namespaces/openshift-config/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.733467287Z namespaces/openshift-config/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.733477717Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.733530178Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.733671511Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.733917167Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.734238554Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.734434169Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.734620543Z namespaces/openshift-config/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.734653364Z namespaces/openshift-config/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.734709935Z namespaces/openshift-config/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.734760206Z namespaces/openshift-config/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.734876309Z namespaces/openshift-console-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.7349191Z namespaces/openshift-console-operator/openshift-console-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.735063113Z namespaces/openshift-console-operator/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.735094874Z namespaces/openshift-console-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.735171926Z namespaces/openshift-console-operator/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.735193277Z namespaces/openshift-console-operator/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.735286399Z namespaces/openshift-console-operator/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.735405721Z namespaces/openshift-console-operator/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.735507604Z namespaces/openshift-console-operator/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.735555145Z namespaces/openshift-console-operator/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.735592976Z namespaces/openshift-console-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.735647677Z namespaces/openshift-console-operator/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.735664287Z namespaces/openshift-console-operator/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.73575902Z namespaces/openshift-console-operator/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.735805611Z namespaces/openshift-console-operator/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.735844052Z namespaces/openshift-console-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.736001235Z namespaces/openshift-console-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.736102768Z namespaces/openshift-console-operator/coordination.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.736117348Z namespaces/openshift-console-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.736152299Z namespaces/openshift-console-operator/coordination.k8s.io/leases/console-operator-lock.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.736229401Z namespaces/openshift-console-operator/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.736281892Z namespaces/openshift-console-operator/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.736878106Z namespaces/openshift-console-operator/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.736989928Z namespaces/openshift-console-operator/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.737230654Z namespaces/openshift-console-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.737307405Z namespaces/openshift-console-operator/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.737438949Z namespaces/openshift-console-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.737551871Z namespaces/openshift-console-operator/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.737733655Z namespaces/openshift-console-operator/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.737801917Z namespaces/openshift-console-operator/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.737851858Z namespaces/openshift-console-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.738027092Z namespaces/openshift-console-operator/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.738055993Z namespaces/openshift-console-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.738115884Z namespaces/openshift-console-operator/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.738172156Z namespaces/openshift-console-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.738266758Z namespaces/openshift-console-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.738336149Z namespaces/openshift-console-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.738409631Z namespaces/openshift-console-operator/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.738460342Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.738506983Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.738552304Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/cluster-monitoring-prometheus-rules.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.738620766Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.738660757Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/console-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.738716138Z namespaces/openshift-console-operator/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.738746459Z namespaces/openshift-console-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.73880889Z namespaces/openshift-console-operator/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.738827701Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.738851861Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.739278861Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.739451405Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.739592149Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.739797183Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.740114281Z namespaces/openshift-console-operator/pods/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.740127031Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-d7xqf/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.740178542Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-d7xqf/console-operator-584dcb6b54-d7xqf.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.740265804Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-d7xqf/console-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.740274464Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-d7xqf/console-operator/console-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.740284845Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-d7xqf/console-operator/console-operator/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.740819087Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-d7xqf/console-operator/console-operator/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.740835687Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-d7xqf/console-operator/console-operator/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.740886979Z namespaces/openshift-console-operator/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.74093438Z namespaces/openshift-console-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.740990201Z namespaces/openshift-console-operator/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.741031442Z namespaces/openshift-console-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.741092673Z namespaces/openshift-console-user-settings/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.741098293Z namespaces/openshift-console-user-settings/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.741104604Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.741151005Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.741306128Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.741480552Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.741611805Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.74182989Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.742308411Z namespaces/openshift-console/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.742347742Z namespaces/openshift-console/openshift-console.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.742604728Z namespaces/openshift-console/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.74265742Z namespaces/openshift-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.742737831Z namespaces/openshift-console/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.742758982Z namespaces/openshift-console/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.742882685Z namespaces/openshift-console/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.743051709Z namespaces/openshift-console/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.743299654Z namespaces/openshift-console/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.743340755Z namespaces/openshift-console/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.743391426Z namespaces/openshift-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.743463818Z namespaces/openshift-console/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.743506059Z namespaces/openshift-console/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.743582901Z namespaces/openshift-console/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.743640792Z namespaces/openshift-console/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.743684313Z namespaces/openshift-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.743768485Z namespaces/openshift-console/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.743843027Z namespaces/openshift-console/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.743900768Z namespaces/openshift-console/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.744479412Z namespaces/openshift-console/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.744575254Z namespaces/openshift-console/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.74483353Z namespaces/openshift-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.744939052Z namespaces/openshift-console/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.745127527Z namespaces/openshift-console/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.74526422Z namespaces/openshift-console/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.745433804Z namespaces/openshift-console/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.745498295Z namespaces/openshift-console/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.745544546Z namespaces/openshift-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.745623848Z namespaces/openshift-console/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.7457058Z namespaces/openshift-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.745800392Z namespaces/openshift-console/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.745931795Z namespaces/openshift-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.746107779Z namespaces/openshift-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.746216932Z namespaces/openshift-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.746280823Z namespaces/openshift-console/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.746339855Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.746407316Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.74655496Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/console.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.746641682Z namespaces/openshift-console/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.746727504Z namespaces/openshift-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.746802175Z namespaces/openshift-console/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.746824096Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.746921188Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.747320908Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.747503052Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.747671425Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.747963662Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.748197248Z namespaces/openshift-console/pods/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.748210988Z namespaces/openshift-console/pods/console-8496f55b4-wjhbs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.74828455Z namespaces/openshift-console/pods/console-8496f55b4-wjhbs/console-8496f55b4-wjhbs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.748407742Z namespaces/openshift-console/pods/console-8496f55b4-wjhbs/console/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.748419523Z namespaces/openshift-console/pods/console-8496f55b4-wjhbs/console/console/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.748423873Z namespaces/openshift-console/pods/console-8496f55b4-wjhbs/console/console/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.748461614Z namespaces/openshift-console/pods/console-8496f55b4-wjhbs/console/console/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.748611077Z namespaces/openshift-console/pods/console-8496f55b4-wjhbs/console/console/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.748702829Z namespaces/openshift-console/pods/console-8496f55b4-wjhbs/console/console/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.748757921Z namespaces/openshift-console/pods/downloads-7c97dc4555-mbhj4/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.748818452Z namespaces/openshift-console/pods/downloads-7c97dc4555-mbhj4/downloads-7c97dc4555-mbhj4.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.748972246Z namespaces/openshift-console/pods/downloads-7c97dc4555-mbhj4/download-server/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.748994886Z namespaces/openshift-console/pods/downloads-7c97dc4555-mbhj4/download-server/download-server/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.749001376Z namespaces/openshift-console/pods/downloads-7c97dc4555-mbhj4/download-server/download-server/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.749027227Z namespaces/openshift-console/pods/downloads-7c97dc4555-mbhj4/download-server/download-server/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.749462747Z namespaces/openshift-console/pods/downloads-7c97dc4555-mbhj4/download-server/download-server/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.749551749Z namespaces/openshift-console/pods/downloads-7c97dc4555-mbhj4/download-server/download-server/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.74960783Z namespaces/openshift-console/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.749666392Z namespaces/openshift-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.749741073Z namespaces/openshift-console/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.749797425Z namespaces/openshift-console/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.750004699Z namespaces/openshift-controller-manager-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.750070041Z namespaces/openshift-controller-manager-operator/openshift-controller-manager-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.750155243Z namespaces/openshift-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.750214715Z namespaces/openshift-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.750278296Z namespaces/openshift-controller-manager-operator/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.750330027Z namespaces/openshift-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.75044135Z namespaces/openshift-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.750561292Z namespaces/openshift-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.750686115Z namespaces/openshift-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.750749517Z namespaces/openshift-controller-manager-operator/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.750817768Z namespaces/openshift-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.750918881Z namespaces/openshift-controller-manager-operator/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.750996353Z namespaces/openshift-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.751114495Z namespaces/openshift-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.751187007Z namespaces/openshift-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.751259299Z namespaces/openshift-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.751365461Z namespaces/openshift-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.751464053Z namespaces/openshift-controller-manager-operator/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.751520815Z namespaces/openshift-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.751700529Z namespaces/openshift-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.751808331Z namespaces/openshift-controller-manager-operator/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.751941484Z namespaces/openshift-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.752060037Z namespaces/openshift-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.75216631Z namespaces/openshift-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.752331524Z namespaces/openshift-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.752558129Z namespaces/openshift-controller-manager-operator/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.7526185Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.752680361Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.752771334Z namespaces/openshift-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.752819625Z namespaces/openshift-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.75304413Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.753103841Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.753184063Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.753268785Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.753297016Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.753344717Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.753401428Z namespaces/openshift-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.753511711Z namespaces/openshift-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.753550482Z namespaces/openshift-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.753561922Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.753599143Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.753748106Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.753969631Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.754098594Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.754288999Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.754679248Z namespaces/openshift-controller-manager-operator/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.754718379Z namespaces/openshift-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.75477146Z namespaces/openshift-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.7547902Z namespaces/openshift-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.754944264Z namespaces/openshift-controller-manager/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.754974034Z namespaces/openshift-controller-manager/openshift-controller-manager.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.755116678Z namespaces/openshift-controller-manager/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.755138909Z namespaces/openshift-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.75519392Z namespaces/openshift-controller-manager/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.755233321Z namespaces/openshift-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.755315213Z namespaces/openshift-controller-manager/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.755399295Z namespaces/openshift-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.755472376Z namespaces/openshift-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.755558048Z namespaces/openshift-controller-manager/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.755589179Z namespaces/openshift-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.755662881Z namespaces/openshift-controller-manager/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.755686491Z namespaces/openshift-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.755795704Z namespaces/openshift-controller-manager/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.755902536Z namespaces/openshift-controller-manager/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.755928387Z namespaces/openshift-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.756006529Z namespaces/openshift-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.75605631Z namespaces/openshift-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.756266675Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.756298245Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.756392987Z namespaces/openshift-controller-manager/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.756443089Z namespaces/openshift-controller-manager/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.756557731Z namespaces/openshift-controller-manager/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.756623983Z namespaces/openshift-controller-manager/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.756728655Z namespaces/openshift-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.756802007Z namespaces/openshift-controller-manager/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.756903529Z namespaces/openshift-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.757030702Z namespaces/openshift-controller-manager/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.757180606Z namespaces/openshift-controller-manager/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.757253607Z namespaces/openshift-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.757283898Z namespaces/openshift-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.7573458Z namespaces/openshift-controller-manager/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.757401361Z namespaces/openshift-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.757453532Z namespaces/openshift-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.757502873Z namespaces/openshift-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.757585625Z namespaces/openshift-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.757688188Z namespaces/openshift-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.757747289Z namespaces/openshift-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.75779454Z namespaces/openshift-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.757886912Z namespaces/openshift-controller-manager/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.757952534Z namespaces/openshift-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.757991984Z namespaces/openshift-controller-manager/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.758000595Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.758054706Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.75822461Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.758434815Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.758597058Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.758851694Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.759070869Z namespaces/openshift-controller-manager/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.759122681Z namespaces/openshift-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.759191222Z namespaces/openshift-controller-manager/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.759224553Z namespaces/openshift-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.759332035Z namespaces/openshift-dns-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.759390667Z namespaces/openshift-dns-operator/openshift-dns-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.759470329Z namespaces/openshift-dns-operator/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.75950888Z namespaces/openshift-dns-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.759598752Z namespaces/openshift-dns-operator/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.759646843Z namespaces/openshift-dns-operator/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.759735505Z namespaces/openshift-dns-operator/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.759838987Z namespaces/openshift-dns-operator/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.75996055Z namespaces/openshift-dns-operator/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.760030952Z namespaces/openshift-dns-operator/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.760082433Z namespaces/openshift-dns-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.760144534Z namespaces/openshift-dns-operator/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.760189375Z namespaces/openshift-dns-operator/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.760289168Z namespaces/openshift-dns-operator/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.760363439Z namespaces/openshift-dns-operator/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.760414801Z namespaces/openshift-dns-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.760520173Z namespaces/openshift-dns-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.760615425Z namespaces/openshift-dns-operator/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.760676047Z namespaces/openshift-dns-operator/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.76080392Z namespaces/openshift-dns-operator/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.760944543Z namespaces/openshift-dns-operator/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.761022165Z namespaces/openshift-dns-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.761117337Z namespaces/openshift-dns-operator/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.761195809Z namespaces/openshift-dns-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.761302791Z namespaces/openshift-dns-operator/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.761474855Z namespaces/openshift-dns-operator/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.761551187Z namespaces/openshift-dns-operator/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.761600758Z namespaces/openshift-dns-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.761643419Z namespaces/openshift-dns-operator/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.76168415Z namespaces/openshift-dns-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.761727791Z namespaces/openshift-dns-operator/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.761775672Z namespaces/openshift-dns-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.761871594Z namespaces/openshift-dns-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.761960647Z namespaces/openshift-dns-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.762005217Z namespaces/openshift-dns-operator/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.762048438Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.76210724Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.762184612Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/dns.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.762241813Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.762279674Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/dns-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.762329165Z namespaces/openshift-dns-operator/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.762370316Z namespaces/openshift-dns-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.762418317Z namespaces/openshift-dns-operator/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.762426427Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.762456988Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.762608911Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.763056912Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.763205965Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.76339326Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.763593034Z namespaces/openshift-dns-operator/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.763620855Z namespaces/openshift-dns-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.763690147Z namespaces/openshift-dns-operator/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.763738377Z namespaces/openshift-dns-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.763811169Z namespaces/openshift-dns/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.763879401Z namespaces/openshift-dns/openshift-dns.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.763953502Z namespaces/openshift-dns/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.763993214Z namespaces/openshift-dns/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.764750121Z namespaces/openshift-dns/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.764791152Z namespaces/openshift-dns/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.764979906Z namespaces/openshift-dns/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.765058558Z namespaces/openshift-dns/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.76513898Z namespaces/openshift-dns/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.765182921Z namespaces/openshift-dns/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.765223642Z namespaces/openshift-dns/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.765279503Z namespaces/openshift-dns/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.765323374Z namespaces/openshift-dns/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.765414706Z namespaces/openshift-dns/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.765470538Z namespaces/openshift-dns/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.765519239Z namespaces/openshift-dns/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.765596981Z namespaces/openshift-dns/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.765663672Z namespaces/openshift-dns/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.765702333Z namespaces/openshift-dns/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.765811445Z namespaces/openshift-dns/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.765930028Z namespaces/openshift-dns/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.766165224Z namespaces/openshift-dns/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.766239406Z namespaces/openshift-dns/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.766511792Z namespaces/openshift-dns/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.766639485Z namespaces/openshift-dns/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.766811959Z namespaces/openshift-dns/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.76687031Z namespaces/openshift-dns/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.766950782Z namespaces/openshift-dns/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.767010443Z namespaces/openshift-dns/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.767061605Z namespaces/openshift-dns/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.767139276Z namespaces/openshift-dns/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.767184327Z namespaces/openshift-dns/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.767261419Z namespaces/openshift-dns/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.767348761Z namespaces/openshift-dns/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.767418963Z namespaces/openshift-dns/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.767454834Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.767507345Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.767560496Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/dns-default.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.767640528Z namespaces/openshift-dns/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.767679489Z namespaces/openshift-dns/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.76774456Z namespaces/openshift-dns/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.767755911Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.767781231Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.767988136Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.76817119Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.768316583Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.768518328Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.768993319Z namespaces/openshift-dns/pods/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.76901382Z namespaces/openshift-dns/pods/dns-default-prbcj/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.769065871Z namespaces/openshift-dns/pods/dns-default-prbcj/dns-default-prbcj.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.769190874Z namespaces/openshift-dns/pods/dns-default-prbcj/dns/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.769214684Z namespaces/openshift-dns/pods/dns-default-prbcj/dns/dns/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.769223675Z namespaces/openshift-dns/pods/dns-default-prbcj/dns/dns/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.769273366Z namespaces/openshift-dns/pods/dns-default-prbcj/dns/dns/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.769413459Z namespaces/openshift-dns/pods/dns-default-prbcj/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.769506161Z namespaces/openshift-dns/pods/dns-default-prbcj/dns/dns/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.769553402Z namespaces/openshift-dns/pods/dns-default-prbcj/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.769575103Z namespaces/openshift-dns/pods/dns-default-prbcj/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.769610744Z namespaces/openshift-dns/pods/dns-default-prbcj/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.769637234Z namespaces/openshift-dns/pods/dns-default-prbcj/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.769761877Z namespaces/openshift-dns/pods/dns-default-prbcj/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.76987327Z namespaces/openshift-dns/pods/dns-default-prbcj/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.769930321Z namespaces/openshift-dns/pods/dns-default-pvns7/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.769987022Z namespaces/openshift-dns/pods/dns-default-pvns7/dns-default-pvns7.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.770077994Z namespaces/openshift-dns/pods/dns-default-pvns7/dns/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.770088985Z namespaces/openshift-dns/pods/dns-default-pvns7/dns/dns/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.770101075Z namespaces/openshift-dns/pods/dns-default-pvns7/dns/dns/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.770157726Z namespaces/openshift-dns/pods/dns-default-pvns7/dns/dns/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.770271769Z namespaces/openshift-dns/pods/dns-default-pvns7/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.770360941Z namespaces/openshift-dns/pods/dns-default-pvns7/dns/dns/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.770422232Z namespaces/openshift-dns/pods/dns-default-pvns7/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.770432783Z namespaces/openshift-dns/pods/dns-default-pvns7/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.770461873Z namespaces/openshift-dns/pods/dns-default-pvns7/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.770502164Z namespaces/openshift-dns/pods/dns-default-pvns7/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.770615697Z namespaces/openshift-dns/pods/dns-default-pvns7/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.770715679Z namespaces/openshift-dns/pods/dns-default-pvns7/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.77076569Z namespaces/openshift-dns/pods/dns-default-tzvrv/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.770813041Z namespaces/openshift-dns/pods/dns-default-tzvrv/dns-default-tzvrv.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.770941084Z namespaces/openshift-dns/pods/dns-default-tzvrv/dns/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.770955795Z namespaces/openshift-dns/pods/dns-default-tzvrv/dns/dns/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.770964165Z namespaces/openshift-dns/pods/dns-default-tzvrv/dns/dns/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.771014746Z namespaces/openshift-dns/pods/dns-default-tzvrv/dns/dns/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.771128399Z namespaces/openshift-dns/pods/dns-default-tzvrv/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.771214401Z namespaces/openshift-dns/pods/dns-default-tzvrv/dns/dns/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.771247511Z namespaces/openshift-dns/pods/dns-default-tzvrv/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.771257532Z namespaces/openshift-dns/pods/dns-default-tzvrv/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.771292452Z namespaces/openshift-dns/pods/dns-default-tzvrv/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.771341764Z namespaces/openshift-dns/pods/dns-default-tzvrv/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.771467856Z namespaces/openshift-dns/pods/dns-default-tzvrv/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.771547628Z namespaces/openshift-dns/pods/dns-default-tzvrv/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.7716048Z namespaces/openshift-dns/pods/node-resolver-5brb2/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.77162081Z namespaces/openshift-dns/pods/node-resolver-5brb2/node-resolver-5brb2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.771747453Z namespaces/openshift-dns/pods/node-resolver-5brb2/dns-node-resolver/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.771759733Z namespaces/openshift-dns/pods/node-resolver-5brb2/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.771769303Z namespaces/openshift-dns/pods/node-resolver-5brb2/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.771813715Z namespaces/openshift-dns/pods/node-resolver-5brb2/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.771944808Z namespaces/openshift-dns/pods/node-resolver-5brb2/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.77203338Z namespaces/openshift-dns/pods/node-resolver-5brb2/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.772103201Z namespaces/openshift-dns/pods/node-resolver-6kcft/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.772155372Z namespaces/openshift-dns/pods/node-resolver-6kcft/node-resolver-6kcft.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.772287135Z namespaces/openshift-dns/pods/node-resolver-6kcft/dns-node-resolver/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.772298396Z namespaces/openshift-dns/pods/node-resolver-6kcft/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.772303786Z namespaces/openshift-dns/pods/node-resolver-6kcft/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.772430169Z namespaces/openshift-dns/pods/node-resolver-6kcft/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.772539161Z namespaces/openshift-dns/pods/node-resolver-6kcft/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.772616123Z namespaces/openshift-dns/pods/node-resolver-6kcft/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.772665314Z namespaces/openshift-dns/pods/node-resolver-kt2nr/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.772713095Z namespaces/openshift-dns/pods/node-resolver-kt2nr/node-resolver-kt2nr.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.772799117Z namespaces/openshift-dns/pods/node-resolver-kt2nr/dns-node-resolver/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.772809447Z namespaces/openshift-dns/pods/node-resolver-kt2nr/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.772814718Z namespaces/openshift-dns/pods/node-resolver-kt2nr/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.772875549Z namespaces/openshift-dns/pods/node-resolver-kt2nr/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.772991822Z namespaces/openshift-dns/pods/node-resolver-kt2nr/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.773076104Z namespaces/openshift-dns/pods/node-resolver-kt2nr/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.773120565Z namespaces/openshift-dns/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.773169256Z namespaces/openshift-dns/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.773227197Z namespaces/openshift-dns/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.773277469Z namespaces/openshift-dns/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.773385031Z namespaces/openshift-etcd/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.773447462Z namespaces/openshift-etcd/openshift-etcd.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.773508644Z namespaces/openshift-etcd/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.773562395Z namespaces/openshift-etcd/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.773632727Z namespaces/openshift-etcd/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.773689118Z namespaces/openshift-etcd/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.7737862Z namespaces/openshift-etcd/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.773915223Z namespaces/openshift-etcd/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.774016246Z namespaces/openshift-etcd/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.774069507Z namespaces/openshift-etcd/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.774119838Z namespaces/openshift-etcd/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.774175199Z namespaces/openshift-etcd/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.774234731Z namespaces/openshift-etcd/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.774333193Z namespaces/openshift-etcd/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.774392514Z namespaces/openshift-etcd/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.774447656Z namespaces/openshift-etcd/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.774540548Z namespaces/openshift-etcd/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.77464473Z namespaces/openshift-etcd/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.774695001Z namespaces/openshift-etcd/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.774814444Z namespaces/openshift-etcd/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.774932547Z namespaces/openshift-etcd/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.775032749Z namespaces/openshift-etcd/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.775120751Z namespaces/openshift-etcd/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.775222654Z namespaces/openshift-etcd/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.775378797Z namespaces/openshift-etcd/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.775578332Z namespaces/openshift-etcd/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.775653964Z namespaces/openshift-etcd/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.775712045Z namespaces/openshift-etcd/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.775774026Z namespaces/openshift-etcd/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.775816907Z namespaces/openshift-etcd/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.77591041Z namespaces/openshift-etcd/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.77595053Z namespaces/openshift-etcd/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.776058763Z namespaces/openshift-etcd/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.776158785Z namespaces/openshift-etcd/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.776228267Z namespaces/openshift-etcd/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.776285348Z namespaces/openshift-etcd/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.776341049Z namespaces/openshift-etcd/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.77638494Z namespaces/openshift-etcd/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.776445052Z namespaces/openshift-etcd/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.776465332Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.776520764Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.776704788Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.777107067Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.777259521Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.777491276Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.777717541Z namespaces/openshift-etcd/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.777763892Z namespaces/openshift-etcd/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.777834834Z namespaces/openshift-etcd/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.777906106Z namespaces/openshift-etcd/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.778039759Z namespaces/openshift-host-network/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.77809654Z namespaces/openshift-host-network/openshift-host-network.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.778165092Z namespaces/openshift-host-network/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.778207923Z namespaces/openshift-host-network/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.778296025Z namespaces/openshift-host-network/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.778353106Z namespaces/openshift-host-network/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.778449248Z namespaces/openshift-host-network/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.778555441Z namespaces/openshift-host-network/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.778656603Z namespaces/openshift-host-network/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.778731275Z namespaces/openshift-host-network/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.778785626Z namespaces/openshift-host-network/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.778852007Z namespaces/openshift-host-network/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.778932469Z namespaces/openshift-host-network/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.779035832Z namespaces/openshift-host-network/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.779103013Z namespaces/openshift-host-network/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.779153814Z namespaces/openshift-host-network/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.779255567Z namespaces/openshift-host-network/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.779353989Z namespaces/openshift-host-network/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.77940171Z namespaces/openshift-host-network/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.779571884Z namespaces/openshift-host-network/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.779678057Z namespaces/openshift-host-network/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.779793179Z namespaces/openshift-host-network/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.779918642Z namespaces/openshift-host-network/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.780021145Z namespaces/openshift-host-network/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.780206429Z namespaces/openshift-host-network/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.780329552Z namespaces/openshift-host-network/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.780389843Z namespaces/openshift-host-network/core/resourcequotas/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.780443854Z namespaces/openshift-host-network/core/resourcequotas/host-network-namespace-quotas.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.780523476Z namespaces/openshift-host-network/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.780582698Z namespaces/openshift-host-network/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.780652599Z namespaces/openshift-host-network/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.780715081Z namespaces/openshift-host-network/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.780791182Z namespaces/openshift-host-network/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.780880874Z namespaces/openshift-host-network/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.780972407Z namespaces/openshift-host-network/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.781278464Z namespaces/openshift-host-network/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.781324515Z namespaces/openshift-host-network/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.781415367Z namespaces/openshift-host-network/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.781447958Z namespaces/openshift-host-network/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.781488919Z namespaces/openshift-host-network/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.78153481Z namespaces/openshift-host-network/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.78156221Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.781611882Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.781779595Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.782004851Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.782137784Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.782330548Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.782529293Z namespaces/openshift-host-network/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.782571284Z namespaces/openshift-host-network/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.782627015Z namespaces/openshift-host-network/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.782656315Z namespaces/openshift-host-network/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.782753198Z namespaces/openshift-image-registry/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.782805619Z namespaces/openshift-image-registry/openshift-image-registry.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.78285341Z namespaces/openshift-image-registry/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.782926302Z namespaces/openshift-image-registry/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.782983423Z namespaces/openshift-image-registry/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.783016634Z namespaces/openshift-image-registry/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.783133377Z namespaces/openshift-image-registry/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.78327526Z namespaces/openshift-image-registry/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.783482065Z namespaces/openshift-image-registry/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.783499475Z namespaces/openshift-image-registry/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.783555227Z namespaces/openshift-image-registry/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.783591167Z namespaces/openshift-image-registry/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.783637418Z namespaces/openshift-image-registry/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.783747271Z namespaces/openshift-image-registry/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.783809532Z namespaces/openshift-image-registry/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.783879644Z namespaces/openshift-image-registry/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.783965836Z namespaces/openshift-image-registry/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.784013447Z namespaces/openshift-image-registry/coordination.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.784021827Z namespaces/openshift-image-registry/coordination.k8s.io/leases/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.784054308Z namespaces/openshift-image-registry/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.78413586Z namespaces/openshift-image-registry/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.784202551Z namespaces/openshift-image-registry/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.785145893Z namespaces/openshift-image-registry/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.785258816Z namespaces/openshift-image-registry/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.785513642Z namespaces/openshift-image-registry/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.785586403Z namespaces/openshift-image-registry/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.78588502Z namespaces/openshift-image-registry/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.786193197Z namespaces/openshift-image-registry/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.786369772Z namespaces/openshift-image-registry/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.786425383Z namespaces/openshift-image-registry/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.786503085Z namespaces/openshift-image-registry/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.786550726Z namespaces/openshift-image-registry/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.786575616Z namespaces/openshift-image-registry/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.786646808Z namespaces/openshift-image-registry/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.786690329Z namespaces/openshift-image-registry/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.786773231Z namespaces/openshift-image-registry/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.786847472Z namespaces/openshift-image-registry/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.786953325Z namespaces/openshift-image-registry/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.786989016Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.787050597Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.787097218Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-operator-alerts.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.787190871Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-rules.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.787281073Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/imagestreams-rules.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.787339084Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.787384995Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/image-registry.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.787434726Z namespaces/openshift-image-registry/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.787473147Z namespaces/openshift-image-registry/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.787510568Z namespaces/openshift-image-registry/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.787533459Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.78759481Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.787768834Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.787984539Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.788116612Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.788318447Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.788498811Z namespaces/openshift-image-registry/pods/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.788510541Z namespaces/openshift-image-registry/pods/image-registry-5f6478cfcf-8vjv8/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.788528671Z namespaces/openshift-image-registry/pods/image-registry-5f6478cfcf-8vjv8/image-registry-5f6478cfcf-8vjv8.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.788635254Z namespaces/openshift-image-registry/pods/image-registry-5f6478cfcf-8vjv8/registry/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.788648884Z namespaces/openshift-image-registry/pods/image-registry-5f6478cfcf-8vjv8/registry/registry/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.788667925Z namespaces/openshift-image-registry/pods/image-registry-5f6478cfcf-8vjv8/registry/registry/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.788704746Z namespaces/openshift-image-registry/pods/image-registry-5f6478cfcf-8vjv8/registry/registry/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.790781294Z namespaces/openshift-image-registry/pods/image-registry-5f6478cfcf-8vjv8/registry/registry/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.790848035Z namespaces/openshift-image-registry/pods/image-registry-5f6478cfcf-8vjv8/registry/registry/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.790883156Z namespaces/openshift-image-registry/pods/node-ca-2hn7j/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.790936597Z namespaces/openshift-image-registry/pods/node-ca-2hn7j/node-ca-2hn7j.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.791012089Z namespaces/openshift-image-registry/pods/node-ca-2hn7j/node-ca/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.791033529Z namespaces/openshift-image-registry/pods/node-ca-2hn7j/node-ca/node-ca/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.79103791Z namespaces/openshift-image-registry/pods/node-ca-2hn7j/node-ca/node-ca/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.79104727Z namespaces/openshift-image-registry/pods/node-ca-2hn7j/node-ca/node-ca/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.791215214Z namespaces/openshift-image-registry/pods/node-ca-2hn7j/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.791281275Z namespaces/openshift-image-registry/pods/node-ca-2hn7j/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.791317326Z namespaces/openshift-image-registry/pods/node-ca-5hgbt/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.791334316Z namespaces/openshift-image-registry/pods/node-ca-5hgbt/node-ca-5hgbt.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.791420789Z namespaces/openshift-image-registry/pods/node-ca-5hgbt/node-ca/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.791432599Z namespaces/openshift-image-registry/pods/node-ca-5hgbt/node-ca/node-ca/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.791442099Z namespaces/openshift-image-registry/pods/node-ca-5hgbt/node-ca/node-ca/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.79148307Z namespaces/openshift-image-registry/pods/node-ca-5hgbt/node-ca/node-ca/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.791665104Z namespaces/openshift-image-registry/pods/node-ca-5hgbt/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.791722385Z namespaces/openshift-image-registry/pods/node-ca-5hgbt/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.791759476Z namespaces/openshift-image-registry/pods/node-ca-dk4md/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.791789007Z namespaces/openshift-image-registry/pods/node-ca-dk4md/node-ca-dk4md.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.791883889Z namespaces/openshift-image-registry/pods/node-ca-dk4md/node-ca/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.79190766Z namespaces/openshift-image-registry/pods/node-ca-dk4md/node-ca/node-ca/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.79192255Z namespaces/openshift-image-registry/pods/node-ca-dk4md/node-ca/node-ca/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.791977842Z namespaces/openshift-image-registry/pods/node-ca-dk4md/node-ca/node-ca/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.792145385Z namespaces/openshift-image-registry/pods/node-ca-dk4md/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.792207457Z namespaces/openshift-image-registry/pods/node-ca-dk4md/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.792225407Z namespaces/openshift-image-registry/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.792273428Z namespaces/openshift-image-registry/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.79234212Z namespaces/openshift-image-registry/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.792372071Z namespaces/openshift-image-registry/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.793901496Z namespaces/openshift-infra/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.793921967Z namespaces/openshift-infra/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.793928007Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.793933317Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.793944397Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.793949097Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.793957637Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.793962327Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.793966828Z namespaces/openshift-ingress-canary/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.793971578Z namespaces/openshift-ingress-canary/openshift-ingress-canary.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.793976228Z namespaces/openshift-ingress-canary/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.793980758Z namespaces/openshift-ingress-canary/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.793993518Z namespaces/openshift-ingress-canary/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.794001438Z namespaces/openshift-ingress-canary/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.795106094Z namespaces/openshift-ingress-canary/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.795124734Z namespaces/openshift-ingress-canary/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.795159505Z namespaces/openshift-ingress-canary/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.795164805Z namespaces/openshift-ingress-canary/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.795170425Z namespaces/openshift-ingress-canary/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.795175616Z namespaces/openshift-ingress-canary/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.795181085Z namespaces/openshift-ingress-canary/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.795186356Z namespaces/openshift-ingress-canary/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.795191386Z namespaces/openshift-ingress-canary/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.795196246Z namespaces/openshift-ingress-canary/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.795201156Z namespaces/openshift-ingress-canary/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.795205996Z namespaces/openshift-ingress-canary/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.795210926Z namespaces/openshift-ingress-canary/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.795216096Z namespaces/openshift-ingress-canary/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.795229347Z namespaces/openshift-ingress-canary/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.79537308Z namespaces/openshift-ingress-canary/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.795442082Z namespaces/openshift-ingress-canary/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.795631996Z namespaces/openshift-ingress-canary/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.795781299Z namespaces/openshift-ingress-canary/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.796088087Z namespaces/openshift-ingress-canary/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.796153188Z namespaces/openshift-ingress-canary/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.796200709Z namespaces/openshift-ingress-canary/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.796271301Z namespaces/openshift-ingress-canary/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.796310622Z namespaces/openshift-ingress-canary/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.796407894Z namespaces/openshift-ingress-canary/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.796456755Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.796550207Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.796631979Z namespaces/openshift-ingress-canary/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.79668017Z namespaces/openshift-ingress-canary/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.796713541Z namespaces/openshift-ingress-canary/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.796762482Z namespaces/openshift-ingress-canary/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.796804893Z namespaces/openshift-ingress-canary/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.796906365Z namespaces/openshift-ingress-canary/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.796929776Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.796965057Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.797433108Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.797639113Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.797808346Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.798076383Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.798528843Z namespaces/openshift-ingress-canary/pods/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.798551674Z namespaces/openshift-ingress-canary/pods/ingress-canary-6tmv2/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.798602815Z namespaces/openshift-ingress-canary/pods/ingress-canary-6tmv2/ingress-canary-6tmv2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.798718287Z namespaces/openshift-ingress-canary/pods/ingress-canary-6tmv2/serve-healthcheck-canary/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.798734648Z namespaces/openshift-ingress-canary/pods/ingress-canary-6tmv2/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.798743158Z namespaces/openshift-ingress-canary/pods/ingress-canary-6tmv2/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.79881161Z namespaces/openshift-ingress-canary/pods/ingress-canary-6tmv2/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.798946193Z namespaces/openshift-ingress-canary/pods/ingress-canary-6tmv2/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.799036535Z namespaces/openshift-ingress-canary/pods/ingress-canary-6tmv2/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.799087096Z namespaces/openshift-ingress-canary/pods/ingress-canary-9nm5r/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.799133237Z namespaces/openshift-ingress-canary/pods/ingress-canary-9nm5r/ingress-canary-9nm5r.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.79923967Z namespaces/openshift-ingress-canary/pods/ingress-canary-9nm5r/serve-healthcheck-canary/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.79925093Z namespaces/openshift-ingress-canary/pods/ingress-canary-9nm5r/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.79927046Z namespaces/openshift-ingress-canary/pods/ingress-canary-9nm5r/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.799312541Z namespaces/openshift-ingress-canary/pods/ingress-canary-9nm5r/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.799428794Z namespaces/openshift-ingress-canary/pods/ingress-canary-9nm5r/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.799518846Z namespaces/openshift-ingress-canary/pods/ingress-canary-9nm5r/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.799558597Z namespaces/openshift-ingress-canary/pods/ingress-canary-c5glj/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.799610668Z namespaces/openshift-ingress-canary/pods/ingress-canary-c5glj/ingress-canary-c5glj.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.79969302Z namespaces/openshift-ingress-canary/pods/ingress-canary-c5glj/serve-healthcheck-canary/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.79970345Z namespaces/openshift-ingress-canary/pods/ingress-canary-c5glj/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.799725581Z namespaces/openshift-ingress-canary/pods/ingress-canary-c5glj/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.799822893Z namespaces/openshift-ingress-canary/pods/ingress-canary-c5glj/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.800025278Z namespaces/openshift-ingress-canary/pods/ingress-canary-c5glj/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.800098219Z namespaces/openshift-ingress-canary/pods/ingress-canary-c5glj/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.800149781Z namespaces/openshift-ingress-canary/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.800185761Z namespaces/openshift-ingress-canary/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.800256343Z namespaces/openshift-ingress-canary/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.800308774Z namespaces/openshift-ingress-canary/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.800442247Z namespaces/openshift-ingress-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.800501929Z namespaces/openshift-ingress-operator/openshift-ingress-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.800588261Z namespaces/openshift-ingress-operator/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.800634392Z namespaces/openshift-ingress-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.800713604Z namespaces/openshift-ingress-operator/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.800758225Z namespaces/openshift-ingress-operator/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.800914708Z namespaces/openshift-ingress-operator/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.80097369Z namespaces/openshift-ingress-operator/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.801070012Z namespaces/openshift-ingress-operator/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.801147794Z namespaces/openshift-ingress-operator/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.801182155Z namespaces/openshift-ingress-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.801268946Z namespaces/openshift-ingress-operator/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.801303857Z namespaces/openshift-ingress-operator/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.80139686Z namespaces/openshift-ingress-operator/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.801458201Z namespaces/openshift-ingress-operator/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.801497822Z namespaces/openshift-ingress-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.801592004Z namespaces/openshift-ingress-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.801685096Z namespaces/openshift-ingress-operator/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.801721527Z namespaces/openshift-ingress-operator/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.802836503Z namespaces/openshift-ingress-operator/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.802959776Z namespaces/openshift-ingress-operator/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.803107079Z namespaces/openshift-ingress-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.803195551Z namespaces/openshift-ingress-operator/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.803288493Z namespaces/openshift-ingress-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.803450077Z namespaces/openshift-ingress-operator/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.803688433Z namespaces/openshift-ingress-operator/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.803777635Z namespaces/openshift-ingress-operator/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.803816716Z namespaces/openshift-ingress-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.803931558Z namespaces/openshift-ingress-operator/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.803981589Z namespaces/openshift-ingress-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.804054291Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.804064031Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.804116972Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/default-wildcard.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.804203354Z namespaces/openshift-ingress-operator/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.804248806Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.804346428Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.8044437Z namespaces/openshift-ingress-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.804766547Z namespaces/openshift-ingress-operator/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.804808339Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.804972942Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.804992353Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ingress-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.805081515Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.805128656Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ingress-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.805205778Z namespaces/openshift-ingress-operator/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.805249949Z namespaces/openshift-ingress-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.805313Z namespaces/openshift-ingress-operator/operator.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.80532191Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.805368432Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/default.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.805461104Z namespaces/openshift-ingress-operator/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.805476334Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.805537345Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.80571812Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.805964785Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.806133039Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.806390845Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.806690662Z namespaces/openshift-ingress-operator/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.806711312Z namespaces/openshift-ingress-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.806793565Z namespaces/openshift-ingress-operator/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.806818405Z namespaces/openshift-ingress-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.806962908Z namespaces/openshift-ingress/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.806989689Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.807073001Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.807112452Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.807195254Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.807240995Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.807345197Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.807623004Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.807843449Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.807945031Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.807997372Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.808071744Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.808106105Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.808186237Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.808248398Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.808283079Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.808375601Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.808628287Z namespaces/openshift-ingress/coordination.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.808641327Z namespaces/openshift-ingress/coordination.k8s.io/leases/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.808684118Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-ca-openshift-gateway.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.808782891Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-deployment-openshift-gateway.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.808881213Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-status-leader-openshift-gateway.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.808986925Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.809028126Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.809465086Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.809586609Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.810108661Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.810184073Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.810451689Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.810546981Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.810754676Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.810902269Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.81093881Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.811021272Z namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.811030392Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.811068944Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.811145085Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.811198476Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.811270938Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.811307669Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.811394781Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.811473003Z namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.811511014Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.811555435Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.811608996Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.811644377Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/openshift-ai-inference-metrics.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.811714719Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.81177256Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/router-default.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.811835471Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.811899353Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.811949414Z namespaces/openshift-ingress/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.811961424Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.812000105Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.812145298Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.812321162Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.812456785Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.81265322Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.812841085Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.812849645Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-hkbqw/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.812919546Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-hkbqw/istiod-openshift-gateway-5677ccb49b-hkbqw.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.81305661Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-hkbqw/discovery/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.81306583Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-hkbqw/discovery/discovery/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.81307861Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-hkbqw/discovery/discovery/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.813127511Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-hkbqw/discovery/discovery/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.822350305Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-hkbqw/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.822433417Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-hkbqw/discovery/discovery/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.822493178Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.822537049Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.822701103Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c/istio-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.822713433Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c/istio-proxy/istio-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.822717723Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.822768794Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.823913741Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.824030104Z namespaces/openshift-ingress/pods/router-default-57898f8749-pb7sc/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.824079565Z namespaces/openshift-ingress/pods/router-default-57898f8749-pb7sc/router-default-57898f8749-pb7sc.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.824217218Z namespaces/openshift-ingress/pods/router-default-57898f8749-pb7sc/router/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.824253629Z namespaces/openshift-ingress/pods/router-default-57898f8749-pb7sc/router/router/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.824264219Z namespaces/openshift-ingress/pods/router-default-57898f8749-pb7sc/router/router/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.824270049Z namespaces/openshift-ingress/pods/router-default-57898f8749-pb7sc/router/router/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.824402512Z namespaces/openshift-ingress/pods/router-default-57898f8749-pb7sc/router/router/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.824469634Z namespaces/openshift-ingress/pods/router-default-57898f8749-pb7sc/router/router/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.824516655Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.824553116Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.824616937Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.824655798Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.824778041Z namespaces/openshift-insights/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.824824062Z namespaces/openshift-insights/openshift-insights.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.824937054Z namespaces/openshift-insights/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.824989586Z namespaces/openshift-insights/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.825063117Z namespaces/openshift-insights/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.825112209Z namespaces/openshift-insights/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.825221901Z namespaces/openshift-insights/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.825327594Z namespaces/openshift-insights/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.825427276Z namespaces/openshift-insights/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.825468987Z namespaces/openshift-insights/apps/deployments/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.825502308Z namespaces/openshift-insights/apps/deployments/insights-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.825574849Z namespaces/openshift-insights/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.82559931Z namespaces/openshift-insights/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.825670851Z namespaces/openshift-insights/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.825700152Z namespaces/openshift-insights/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.825786384Z namespaces/openshift-insights/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.825850106Z namespaces/openshift-insights/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.825928377Z namespaces/openshift-insights/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.82601293Z namespaces/openshift-insights/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.826081331Z namespaces/openshift-insights/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.826124052Z namespaces/openshift-insights/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.827217047Z namespaces/openshift-insights/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.827292719Z namespaces/openshift-insights/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.827715859Z namespaces/openshift-insights/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.827832082Z namespaces/openshift-insights/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.828145709Z namespaces/openshift-insights/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.828252241Z namespaces/openshift-insights/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.828403065Z namespaces/openshift-insights/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.828463046Z namespaces/openshift-insights/core/configmaps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.828516757Z namespaces/openshift-insights/core/configmaps/service-ca-bundle.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.828577649Z namespaces/openshift-insights/core/serviceaccounts/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.82861706Z namespaces/openshift-insights/core/serviceaccounts/gather.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.828702812Z namespaces/openshift-insights/core/serviceaccounts/operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.828761233Z namespaces/openshift-insights/core/services/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.828790854Z namespaces/openshift-insights/core/services/metrics.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.828867636Z namespaces/openshift-insights/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.828939757Z namespaces/openshift-insights/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.828991679Z namespaces/openshift-insights/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.8290341Z namespaces/openshift-insights/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.829097701Z namespaces/openshift-insights/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.829129722Z namespaces/openshift-insights/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.829214784Z namespaces/openshift-insights/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.829292846Z namespaces/openshift-insights/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.829346877Z namespaces/openshift-insights/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.829387318Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.829456319Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.82948864Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/insights-prometheus-rules.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.829572782Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.829589342Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/insights-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.829658614Z namespaces/openshift-insights/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.829704785Z namespaces/openshift-insights/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.829769437Z namespaces/openshift-insights/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.829795317Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.829843258Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.830108774Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.830302769Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.830459853Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.830676867Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.830936324Z namespaces/openshift-insights/pods/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.830954454Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-2vf8b/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.830961694Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-2vf8b/insights-operator-5885fdb759-2vf8b.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.831070376Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-2vf8b/insights-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.831079117Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-2vf8b/insights-operator/insights-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.831085607Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-2vf8b/insights-operator/insights-operator/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.831138218Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-2vf8b/insights-operator/insights-operator/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.831545058Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-2vf8b/insights-operator/insights-operator/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.831872225Z namespaces/openshift-insights/pods/insights-runtime-extractor-gjcfv/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.831908316Z namespaces/openshift-insights/pods/insights-runtime-extractor-gjcfv/insights-runtime-extractor-gjcfv.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.832028979Z namespaces/openshift-insights/pods/insights-runtime-extractor-gjcfv/exporter/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.832041409Z namespaces/openshift-insights/pods/insights-runtime-extractor-gjcfv/exporter/exporter/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.832051399Z namespaces/openshift-insights/pods/insights-runtime-extractor-gjcfv/exporter/exporter/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.83207343Z namespaces/openshift-insights/pods/insights-runtime-extractor-gjcfv/exporter/exporter/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.832173972Z namespaces/openshift-insights/pods/insights-runtime-extractor-gjcfv/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.832245904Z namespaces/openshift-insights/pods/insights-runtime-extractor-gjcfv/exporter/exporter/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.832283795Z namespaces/openshift-insights/pods/insights-runtime-extractor-gjcfv/extractor/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.832295525Z namespaces/openshift-insights/pods/insights-runtime-extractor-gjcfv/extractor/extractor/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.832301655Z namespaces/openshift-insights/pods/insights-runtime-extractor-gjcfv/extractor/extractor/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.832341346Z namespaces/openshift-insights/pods/insights-runtime-extractor-gjcfv/extractor/extractor/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.832469669Z namespaces/openshift-insights/pods/insights-runtime-extractor-gjcfv/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.832533501Z namespaces/openshift-insights/pods/insights-runtime-extractor-gjcfv/extractor/extractor/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.832574611Z namespaces/openshift-insights/pods/insights-runtime-extractor-gjcfv/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.832596322Z namespaces/openshift-insights/pods/insights-runtime-extractor-gjcfv/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.832605842Z namespaces/openshift-insights/pods/insights-runtime-extractor-gjcfv/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.832647693Z namespaces/openshift-insights/pods/insights-runtime-extractor-gjcfv/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.832736345Z namespaces/openshift-insights/pods/insights-runtime-extractor-gjcfv/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.832812077Z namespaces/openshift-insights/pods/insights-runtime-extractor-gjcfv/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.832849388Z namespaces/openshift-insights/pods/insights-runtime-extractor-l6rw6/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.832927279Z namespaces/openshift-insights/pods/insights-runtime-extractor-l6rw6/insights-runtime-extractor-l6rw6.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.833020752Z namespaces/openshift-insights/pods/insights-runtime-extractor-l6rw6/exporter/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.833030572Z namespaces/openshift-insights/pods/insights-runtime-extractor-l6rw6/exporter/exporter/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.833048242Z namespaces/openshift-insights/pods/insights-runtime-extractor-l6rw6/exporter/exporter/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.833068053Z namespaces/openshift-insights/pods/insights-runtime-extractor-l6rw6/exporter/exporter/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.833237307Z namespaces/openshift-insights/pods/insights-runtime-extractor-l6rw6/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.833296918Z namespaces/openshift-insights/pods/insights-runtime-extractor-l6rw6/exporter/exporter/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.833330269Z namespaces/openshift-insights/pods/insights-runtime-extractor-l6rw6/extractor/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.833341479Z namespaces/openshift-insights/pods/insights-runtime-extractor-l6rw6/extractor/extractor/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.833358649Z namespaces/openshift-insights/pods/insights-runtime-extractor-l6rw6/extractor/extractor/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.833411811Z namespaces/openshift-insights/pods/insights-runtime-extractor-l6rw6/extractor/extractor/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.833518373Z namespaces/openshift-insights/pods/insights-runtime-extractor-l6rw6/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.833581275Z namespaces/openshift-insights/pods/insights-runtime-extractor-l6rw6/extractor/extractor/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.833616035Z namespaces/openshift-insights/pods/insights-runtime-extractor-l6rw6/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.833631626Z namespaces/openshift-insights/pods/insights-runtime-extractor-l6rw6/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.833653016Z namespaces/openshift-insights/pods/insights-runtime-extractor-l6rw6/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.833751399Z namespaces/openshift-insights/pods/insights-runtime-extractor-l6rw6/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.833887682Z namespaces/openshift-insights/pods/insights-runtime-extractor-l6rw6/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.833990784Z namespaces/openshift-insights/pods/insights-runtime-extractor-l6rw6/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.834012275Z namespaces/openshift-insights/pods/insights-runtime-extractor-mz5cv/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.834060376Z namespaces/openshift-insights/pods/insights-runtime-extractor-mz5cv/insights-runtime-extractor-mz5cv.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.834148988Z namespaces/openshift-insights/pods/insights-runtime-extractor-mz5cv/exporter/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.834161218Z namespaces/openshift-insights/pods/insights-runtime-extractor-mz5cv/exporter/exporter/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.834167368Z namespaces/openshift-insights/pods/insights-runtime-extractor-mz5cv/exporter/exporter/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.834193959Z namespaces/openshift-insights/pods/insights-runtime-extractor-mz5cv/exporter/exporter/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.834303562Z namespaces/openshift-insights/pods/insights-runtime-extractor-mz5cv/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.834380633Z namespaces/openshift-insights/pods/insights-runtime-extractor-mz5cv/exporter/exporter/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.834416244Z namespaces/openshift-insights/pods/insights-runtime-extractor-mz5cv/extractor/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.834424724Z namespaces/openshift-insights/pods/insights-runtime-extractor-mz5cv/extractor/extractor/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.834435905Z namespaces/openshift-insights/pods/insights-runtime-extractor-mz5cv/extractor/extractor/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.834488396Z namespaces/openshift-insights/pods/insights-runtime-extractor-mz5cv/extractor/extractor/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.834579218Z namespaces/openshift-insights/pods/insights-runtime-extractor-mz5cv/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.83465711Z namespaces/openshift-insights/pods/insights-runtime-extractor-mz5cv/extractor/extractor/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.83468137Z namespaces/openshift-insights/pods/insights-runtime-extractor-mz5cv/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.83468754Z namespaces/openshift-insights/pods/insights-runtime-extractor-mz5cv/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.83469408Z namespaces/openshift-insights/pods/insights-runtime-extractor-mz5cv/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.834746662Z namespaces/openshift-insights/pods/insights-runtime-extractor-mz5cv/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.834829274Z namespaces/openshift-insights/pods/insights-runtime-extractor-mz5cv/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.834922616Z namespaces/openshift-insights/pods/insights-runtime-extractor-mz5cv/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.834962497Z namespaces/openshift-insights/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.834992217Z namespaces/openshift-insights/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.835052599Z namespaces/openshift-insights/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.83508474Z namespaces/openshift-insights/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.835141501Z namespaces/openshift-keda/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.835155451Z namespaces/openshift-keda/coordination.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.835161561Z namespaces/openshift-keda/coordination.k8s.io/leases/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.835190892Z namespaces/openshift-keda/coordination.k8s.io/leases/olm-operator.keda.sh.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.835282574Z namespaces/openshift-keda/coordination.k8s.io/leases/operator.keda.sh.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.835315865Z namespaces/openshift-keda/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.835322895Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.835363026Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/keda-olm-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.835418107Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.835451198Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-admission-webhooks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.83554704Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-metrics-apiserver.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.835623992Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.835682534Z namespaces/openshift-keda/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.835695524Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.835725154Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.835939299Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.836120473Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/custom-metrics-autoscaler.v2.19.0-2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.836323228Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.836535813Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.836675766Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.83683194Z namespaces/openshift-keda/operators.coreos.com/installplans/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.836885701Z namespaces/openshift-keda/operators.coreos.com/installplans/install-vxlw6.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.837030354Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.837048065Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/custom-metrics-autoscaler.v2.19.0-2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.837110296Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.837138427Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/openshift-keda.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.837218089Z namespaces/openshift-keda/operators.coreos.com/subscriptions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.837234289Z namespaces/openshift-keda/operators.coreos.com/subscriptions/openshift-custom-metrics-autoscaler-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.837332181Z namespaces/openshift-kube-apiserver-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.837370062Z namespaces/openshift-kube-apiserver-operator/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.837444064Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.837468395Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.837518936Z namespaces/openshift-kube-apiserver-operator/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.837556317Z namespaces/openshift-kube-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.837637279Z namespaces/openshift-kube-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.83770844Z namespaces/openshift-kube-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.837798283Z namespaces/openshift-kube-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.837847644Z namespaces/openshift-kube-apiserver-operator/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.837935786Z namespaces/openshift-kube-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.837987197Z namespaces/openshift-kube-apiserver-operator/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.838008247Z namespaces/openshift-kube-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.838102919Z namespaces/openshift-kube-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.838145271Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.838185861Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.838266363Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.838320975Z namespaces/openshift-kube-apiserver-operator/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.838355715Z namespaces/openshift-kube-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.838468298Z namespaces/openshift-kube-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.83855813Z namespaces/openshift-kube-apiserver-operator/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.838652602Z namespaces/openshift-kube-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.838730994Z namespaces/openshift-kube-apiserver-operator/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.838807236Z namespaces/openshift-kube-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.838966539Z namespaces/openshift-kube-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.839171894Z namespaces/openshift-kube-apiserver-operator/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.839461921Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.839493262Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.839544793Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.839582094Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.839654775Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.839700916Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.839811809Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.839941092Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.839999163Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.840050944Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.840117696Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.840169567Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.840236119Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.840260309Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.84028902Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.840447064Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.840623268Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.840749951Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.840985836Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.8411658Z namespaces/openshift-kube-apiserver-operator/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.841198211Z namespaces/openshift-kube-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.841261303Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.841289463Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.841354305Z namespaces/openshift-kube-apiserver/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.841384125Z namespaces/openshift-kube-apiserver/openshift-kube-apiserver.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.841445317Z namespaces/openshift-kube-apiserver/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.841475937Z namespaces/openshift-kube-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.841539589Z namespaces/openshift-kube-apiserver/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.84157207Z namespaces/openshift-kube-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.841665672Z namespaces/openshift-kube-apiserver/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.841733194Z namespaces/openshift-kube-apiserver/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.841810375Z namespaces/openshift-kube-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.841874087Z namespaces/openshift-kube-apiserver/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.841951239Z namespaces/openshift-kube-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.84202436Z namespaces/openshift-kube-apiserver/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.842045371Z namespaces/openshift-kube-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.842139603Z namespaces/openshift-kube-apiserver/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.842183734Z namespaces/openshift-kube-apiserver/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.842229475Z namespaces/openshift-kube-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.842311447Z namespaces/openshift-kube-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.842363518Z namespaces/openshift-kube-apiserver/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.842409289Z namespaces/openshift-kube-apiserver/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.842495591Z namespaces/openshift-kube-apiserver/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.842574823Z namespaces/openshift-kube-apiserver/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.842669155Z namespaces/openshift-kube-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.842742817Z namespaces/openshift-kube-apiserver/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.842827749Z namespaces/openshift-kube-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.843003763Z namespaces/openshift-kube-apiserver/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.843163116Z namespaces/openshift-kube-apiserver/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.843215878Z namespaces/openshift-kube-apiserver/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.843248989Z namespaces/openshift-kube-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.84330939Z namespaces/openshift-kube-apiserver/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.84332685Z namespaces/openshift-kube-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.843392472Z namespaces/openshift-kube-apiserver/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.843432113Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.843509905Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.843588637Z namespaces/openshift-kube-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.843647918Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.843664868Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.84374599Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.843765961Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/api-usage.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.843889404Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/podsecurity.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.843979046Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.844016286Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/openshift-kube-apiserver.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.844079358Z namespaces/openshift-kube-apiserver/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.844100008Z namespaces/openshift-kube-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.84416168Z namespaces/openshift-kube-apiserver/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.84417931Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.844236752Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.844387125Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.844564119Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.844682882Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.844900477Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.845096781Z namespaces/openshift-kube-apiserver/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.845219854Z namespaces/openshift-kube-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.845234684Z namespaces/openshift-kube-apiserver/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.845241875Z namespaces/openshift-kube-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.845317637Z namespaces/openshift-kube-controller-manager-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.845361467Z namespaces/openshift-kube-controller-manager-operator/openshift-kube-controller-manager-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.845416829Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.84546229Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.845535702Z namespaces/openshift-kube-controller-manager-operator/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.845582313Z namespaces/openshift-kube-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.845652304Z namespaces/openshift-kube-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.845734236Z namespaces/openshift-kube-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.845807628Z namespaces/openshift-kube-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.84588352Z namespaces/openshift-kube-controller-manager-operator/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.84592274Z namespaces/openshift-kube-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.845997622Z namespaces/openshift-kube-controller-manager-operator/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.846016643Z namespaces/openshift-kube-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.846102285Z namespaces/openshift-kube-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.846159696Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.846192907Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.846275969Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.84634706Z namespaces/openshift-kube-controller-manager-operator/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.846374261Z namespaces/openshift-kube-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.846505434Z namespaces/openshift-kube-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.846582516Z namespaces/openshift-kube-controller-manager-operator/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.846672458Z namespaces/openshift-kube-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.84675531Z namespaces/openshift-kube-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.846829541Z namespaces/openshift-kube-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.846986575Z namespaces/openshift-kube-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.84718794Z namespaces/openshift-kube-controller-manager-operator/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.847289172Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.847324843Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.847387044Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.847428065Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.847490617Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.847522797Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.847607699Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.847682371Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.847739122Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.847764843Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.847830025Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.847904717Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/kube-controller-manager-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.847968168Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.848014559Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.84806022Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.84807093Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.848113691Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.848271245Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.8485008Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.848645894Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.848963861Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.849165826Z namespaces/openshift-kube-controller-manager-operator/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.849203867Z namespaces/openshift-kube-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.849275708Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.849324849Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.849393671Z namespaces/openshift-kube-controller-manager/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.849442522Z namespaces/openshift-kube-controller-manager/openshift-kube-controller-manager.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.849503023Z namespaces/openshift-kube-controller-manager/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.849551384Z namespaces/openshift-kube-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.849616366Z namespaces/openshift-kube-controller-manager/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.849668387Z namespaces/openshift-kube-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.849758759Z namespaces/openshift-kube-controller-manager/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.849840071Z namespaces/openshift-kube-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.850053286Z namespaces/openshift-kube-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.850107437Z namespaces/openshift-kube-controller-manager/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.850152998Z namespaces/openshift-kube-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.8502041Z namespaces/openshift-kube-controller-manager/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.850242671Z namespaces/openshift-kube-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.850324933Z namespaces/openshift-kube-controller-manager/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.850378044Z namespaces/openshift-kube-controller-manager/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.850418244Z namespaces/openshift-kube-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.850519677Z namespaces/openshift-kube-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.850539187Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.850545807Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.850621229Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/cluster-policy-controller-lock.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.850701891Z namespaces/openshift-kube-controller-manager/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.850744292Z namespaces/openshift-kube-controller-manager/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.850852715Z namespaces/openshift-kube-controller-manager/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.850965877Z namespaces/openshift-kube-controller-manager/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.85153195Z namespaces/openshift-kube-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.851607932Z namespaces/openshift-kube-controller-manager/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.851691274Z namespaces/openshift-kube-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.851811697Z namespaces/openshift-kube-controller-manager/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.852000721Z namespaces/openshift-kube-controller-manager/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.852044072Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.852081193Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.852136244Z namespaces/openshift-kube-controller-manager/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.852170105Z namespaces/openshift-kube-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.852229926Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.852257627Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.852345789Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.852421651Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.852474032Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.852501653Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.852555454Z namespaces/openshift-kube-controller-manager/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.852585365Z namespaces/openshift-kube-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.852637426Z namespaces/openshift-kube-controller-manager/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.852645566Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.852689137Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.852831071Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.853057096Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.853190489Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.853384613Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.853571928Z namespaces/openshift-kube-controller-manager/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.853602418Z namespaces/openshift-kube-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.85365603Z namespaces/openshift-kube-controller-manager/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.85369234Z namespaces/openshift-kube-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.853752412Z namespaces/openshift-kube-scheduler-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.853793263Z namespaces/openshift-kube-scheduler-operator/openshift-kube-scheduler-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.853881445Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.853929226Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.853992327Z namespaces/openshift-kube-scheduler-operator/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.854021378Z namespaces/openshift-kube-scheduler-operator/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.85410888Z namespaces/openshift-kube-scheduler-operator/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.854184572Z namespaces/openshift-kube-scheduler-operator/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.854265914Z namespaces/openshift-kube-scheduler-operator/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.854308465Z namespaces/openshift-kube-scheduler-operator/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.854345486Z namespaces/openshift-kube-scheduler-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.854405427Z namespaces/openshift-kube-scheduler-operator/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.854436058Z namespaces/openshift-kube-scheduler-operator/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.854521539Z namespaces/openshift-kube-scheduler-operator/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.854566351Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.854603081Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.854708004Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.854760235Z namespaces/openshift-kube-scheduler-operator/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.854794086Z namespaces/openshift-kube-scheduler-operator/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.854944789Z namespaces/openshift-kube-scheduler-operator/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.855022861Z namespaces/openshift-kube-scheduler-operator/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.855100713Z namespaces/openshift-kube-scheduler-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.855184945Z namespaces/openshift-kube-scheduler-operator/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.855262057Z namespaces/openshift-kube-scheduler-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.85539117Z namespaces/openshift-kube-scheduler-operator/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.855553034Z namespaces/openshift-kube-scheduler-operator/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.855598175Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.855634805Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.855692017Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.855729418Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.855785589Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.85581836Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.855940383Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.856015984Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.856054065Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.856089726Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.856142877Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.856169878Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.856222879Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.856230869Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.85628199Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.856426164Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.856606288Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.856741461Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.856967156Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.857294744Z namespaces/openshift-kube-scheduler-operator/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.857329345Z namespaces/openshift-kube-scheduler-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.857398906Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.857422987Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.857491328Z namespaces/openshift-kube-scheduler/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.8575389Z namespaces/openshift-kube-scheduler/openshift-kube-scheduler.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.857596871Z namespaces/openshift-kube-scheduler/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.857636312Z namespaces/openshift-kube-scheduler/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.857686413Z namespaces/openshift-kube-scheduler/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.857727904Z namespaces/openshift-kube-scheduler/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.857809166Z namespaces/openshift-kube-scheduler/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.857941349Z namespaces/openshift-kube-scheduler/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.858023671Z namespaces/openshift-kube-scheduler/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.858070212Z namespaces/openshift-kube-scheduler/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.858109523Z namespaces/openshift-kube-scheduler/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.858164444Z namespaces/openshift-kube-scheduler/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.858208785Z namespaces/openshift-kube-scheduler/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.858297087Z namespaces/openshift-kube-scheduler/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.858335058Z namespaces/openshift-kube-scheduler/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.858373459Z namespaces/openshift-kube-scheduler/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.858484031Z namespaces/openshift-kube-scheduler/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.858546943Z namespaces/openshift-kube-scheduler/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.858576193Z namespaces/openshift-kube-scheduler/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.858704987Z namespaces/openshift-kube-scheduler/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.858780908Z namespaces/openshift-kube-scheduler/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.858990913Z namespaces/openshift-kube-scheduler/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.859064075Z namespaces/openshift-kube-scheduler/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.859145837Z namespaces/openshift-kube-scheduler/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.859271409Z namespaces/openshift-kube-scheduler/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.859438433Z namespaces/openshift-kube-scheduler/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.859487694Z namespaces/openshift-kube-scheduler/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.859523975Z namespaces/openshift-kube-scheduler/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.859569207Z namespaces/openshift-kube-scheduler/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.859605897Z namespaces/openshift-kube-scheduler/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.859665449Z namespaces/openshift-kube-scheduler/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.85969863Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.859775891Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.859882174Z namespaces/openshift-kube-scheduler/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.859934355Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.859971206Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.860036887Z namespaces/openshift-kube-scheduler/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.860061628Z namespaces/openshift-kube-scheduler/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.860118619Z namespaces/openshift-kube-scheduler/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.860128049Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.86017175Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.860328324Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.860507988Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.860639901Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.860829566Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.861100362Z namespaces/openshift-kube-scheduler/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.861117712Z namespaces/openshift-kube-scheduler/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.861179714Z namespaces/openshift-kube-scheduler/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.861228955Z namespaces/openshift-kube-scheduler/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.861323887Z namespaces/openshift-kube-storage-version-migrator-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.861359698Z namespaces/openshift-kube-storage-version-migrator-operator/openshift-kube-storage-version-migrator-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.861419969Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.861468791Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.861521542Z namespaces/openshift-kube-storage-version-migrator-operator/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.861544442Z namespaces/openshift-kube-storage-version-migrator-operator/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.861638854Z namespaces/openshift-kube-storage-version-migrator-operator/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.861762457Z namespaces/openshift-kube-storage-version-migrator-operator/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.8618945Z namespaces/openshift-kube-storage-version-migrator-operator/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.861949492Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.861997133Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.862050304Z namespaces/openshift-kube-storage-version-migrator-operator/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.862101395Z namespaces/openshift-kube-storage-version-migrator-operator/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.862183417Z namespaces/openshift-kube-storage-version-migrator-operator/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.862240208Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.862261529Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.862349821Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.862398752Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.862409732Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.862450003Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/openshift-kube-storage-version-migrator-operator-lock.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.862507555Z namespaces/openshift-kube-storage-version-migrator-operator/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.862557646Z namespaces/openshift-kube-storage-version-migrator-operator/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.862680779Z namespaces/openshift-kube-storage-version-migrator-operator/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.86277005Z namespaces/openshift-kube-storage-version-migrator-operator/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.862998276Z namespaces/openshift-kube-storage-version-migrator-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.863069617Z namespaces/openshift-kube-storage-version-migrator-operator/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.86319228Z namespaces/openshift-kube-storage-version-migrator-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.863310123Z namespaces/openshift-kube-storage-version-migrator-operator/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.863493697Z namespaces/openshift-kube-storage-version-migrator-operator/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.863540379Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.86361661Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.863679272Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.863701092Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.863764834Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.863803874Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.863926317Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.86403425Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.864076651Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.864130152Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.864164493Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.864201124Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.864258395Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.864273405Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.864318697Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.864577663Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.864769367Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.864954301Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.865163626Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.86535803Z namespaces/openshift-kube-storage-version-migrator-operator/pods/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.865370221Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-jm6np/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.865403222Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-jm6np/kube-storage-version-migrator-operator-6f4d6fb6bc-jm6np.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.865510694Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-jm6np/kube-storage-version-migrator-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.865524024Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-jm6np/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.865532294Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-jm6np/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.865589866Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-jm6np/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.865736809Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-jm6np/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.865884713Z namespaces/openshift-kube-storage-version-migrator-operator/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.865932734Z namespaces/openshift-kube-storage-version-migrator-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.865986005Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.866037316Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.866102818Z namespaces/openshift-kube-storage-version-migrator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.866133719Z namespaces/openshift-kube-storage-version-migrator/openshift-kube-storage-version-migrator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.86620669Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.866223021Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.866284162Z namespaces/openshift-kube-storage-version-migrator/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.866337113Z namespaces/openshift-kube-storage-version-migrator/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.866419225Z namespaces/openshift-kube-storage-version-migrator/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.866528037Z namespaces/openshift-kube-storage-version-migrator/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.86662475Z namespaces/openshift-kube-storage-version-migrator/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.866688781Z namespaces/openshift-kube-storage-version-migrator/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.866724372Z namespaces/openshift-kube-storage-version-migrator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.866788063Z namespaces/openshift-kube-storage-version-migrator/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.866828584Z namespaces/openshift-kube-storage-version-migrator/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.866954147Z namespaces/openshift-kube-storage-version-migrator/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.866998799Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.867045669Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.867133151Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.867177593Z namespaces/openshift-kube-storage-version-migrator/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.867246004Z namespaces/openshift-kube-storage-version-migrator/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.867341576Z namespaces/openshift-kube-storage-version-migrator/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.867419188Z namespaces/openshift-kube-storage-version-migrator/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.867539441Z namespaces/openshift-kube-storage-version-migrator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.867616063Z namespaces/openshift-kube-storage-version-migrator/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.867826078Z namespaces/openshift-kube-storage-version-migrator/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.867979831Z namespaces/openshift-kube-storage-version-migrator/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.868138725Z namespaces/openshift-kube-storage-version-migrator/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.868181546Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.868237597Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.868291629Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.868319439Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.86838198Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.868417911Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.868502563Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.868582115Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.868615966Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.868711418Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.868753799Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.86879161Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.868845521Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.868879132Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.868953774Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.869098967Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.869277081Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.869393674Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.869607489Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.869903856Z namespaces/openshift-kube-storage-version-migrator/pods/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.869933616Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wgpdw/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.869951977Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wgpdw/migrator-6865c9967-wgpdw.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.870055089Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wgpdw/graceful-termination/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.87006683Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wgpdw/graceful-termination/graceful-termination/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.87008084Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wgpdw/graceful-termination/graceful-termination/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.870113061Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wgpdw/graceful-termination/graceful-termination/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.870206223Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wgpdw/graceful-termination/graceful-termination/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.870273324Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wgpdw/graceful-termination/graceful-termination/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.870297705Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wgpdw/migrator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.870310475Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wgpdw/migrator/migrator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.870318235Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wgpdw/migrator/migrator/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.870355336Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wgpdw/migrator/migrator/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.870452139Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wgpdw/migrator/migrator/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.870523Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-wgpdw/migrator/migrator/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.870538421Z namespaces/openshift-kube-storage-version-migrator/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.870577641Z namespaces/openshift-kube-storage-version-migrator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.870652403Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.870671844Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.870740445Z namespaces/openshift-lws-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.870791276Z namespaces/openshift-lws-operator/openshift-lws-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.870872528Z namespaces/openshift-lws-operator/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.870914769Z namespaces/openshift-lws-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.870984121Z namespaces/openshift-lws-operator/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.871002471Z namespaces/openshift-lws-operator/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.871098613Z namespaces/openshift-lws-operator/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.871241657Z namespaces/openshift-lws-operator/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.8713636Z namespaces/openshift-lws-operator/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.871416361Z namespaces/openshift-lws-operator/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.871445071Z namespaces/openshift-lws-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.871501213Z namespaces/openshift-lws-operator/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.871529263Z namespaces/openshift-lws-operator/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.871623406Z namespaces/openshift-lws-operator/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.871653816Z namespaces/openshift-lws-operator/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.871690637Z namespaces/openshift-lws-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.871839141Z namespaces/openshift-lws-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.871913692Z namespaces/openshift-lws-operator/coordination.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.871926243Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.871951713Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/b8b2488c.x-k8s.io.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.872045335Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/openshift-lws-operator-lock.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.872103317Z namespaces/openshift-lws-operator/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.872138478Z namespaces/openshift-lws-operator/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.87225667Z namespaces/openshift-lws-operator/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.872345302Z namespaces/openshift-lws-operator/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.87266467Z namespaces/openshift-lws-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.872736401Z namespaces/openshift-lws-operator/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.872957097Z namespaces/openshift-lws-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.873077029Z namespaces/openshift-lws-operator/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.873296094Z namespaces/openshift-lws-operator/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.873362336Z namespaces/openshift-lws-operator/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.873430877Z namespaces/openshift-lws-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.873500279Z namespaces/openshift-lws-operator/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.87353554Z namespaces/openshift-lws-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.873588131Z namespaces/openshift-lws-operator/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.873626222Z namespaces/openshift-lws-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.873713304Z namespaces/openshift-lws-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.873792166Z namespaces/openshift-lws-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.873845447Z namespaces/openshift-lws-operator/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.873920399Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.87398932Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.874034311Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/lws-controller-manager-metrics-monitor.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.874092483Z namespaces/openshift-lws-operator/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.874127544Z namespaces/openshift-lws-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.874184815Z namespaces/openshift-lws-operator/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.874195785Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.874236336Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.87438915Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.874576134Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.874733767Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.875088006Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.87527938Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.875468235Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.875508225Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/install-qr2xr.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.875590527Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.875621018Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.87568821Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.87572629Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/leader-worker-set.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.875782292Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.875832673Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/leader-worker-set.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.875930455Z namespaces/openshift-lws-operator/pods/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.875953386Z namespaces/openshift-lws-operator/pods/lws-controller-manager-58bd97cf4-7mprj/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.875996767Z namespaces/openshift-lws-operator/pods/lws-controller-manager-58bd97cf4-7mprj/lws-controller-manager-58bd97cf4-7mprj.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.876087759Z namespaces/openshift-lws-operator/pods/lws-controller-manager-58bd97cf4-7mprj/manager/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.876099999Z namespaces/openshift-lws-operator/pods/lws-controller-manager-58bd97cf4-7mprj/manager/manager/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.876110319Z namespaces/openshift-lws-operator/pods/lws-controller-manager-58bd97cf4-7mprj/manager/manager/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.87614642Z namespaces/openshift-lws-operator/pods/lws-controller-manager-58bd97cf4-7mprj/manager/manager/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.876511549Z namespaces/openshift-lws-operator/pods/lws-controller-manager-58bd97cf4-7mprj/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.87657184Z namespaces/openshift-lws-operator/pods/lws-controller-manager-58bd97cf4-7mprj/manager/manager/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.876625881Z namespaces/openshift-lws-operator/pods/lws-controller-manager-58bd97cf4-r6dk4/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.876668882Z namespaces/openshift-lws-operator/pods/lws-controller-manager-58bd97cf4-r6dk4/lws-controller-manager-58bd97cf4-r6dk4.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.876748874Z namespaces/openshift-lws-operator/pods/lws-controller-manager-58bd97cf4-r6dk4/manager/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.876759854Z namespaces/openshift-lws-operator/pods/lws-controller-manager-58bd97cf4-r6dk4/manager/manager/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.876773635Z namespaces/openshift-lws-operator/pods/lws-controller-manager-58bd97cf4-r6dk4/manager/manager/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.876828506Z namespaces/openshift-lws-operator/pods/lws-controller-manager-58bd97cf4-r6dk4/manager/manager/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.87698883Z namespaces/openshift-lws-operator/pods/lws-controller-manager-58bd97cf4-r6dk4/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.877070012Z namespaces/openshift-lws-operator/pods/lws-controller-manager-58bd97cf4-r6dk4/manager/manager/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.877136163Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-dqnrn/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.877182594Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-dqnrn/openshift-lws-operator-fd8ccff4c-dqnrn.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.877285247Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-dqnrn/openshift-lws-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.877295597Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-dqnrn/openshift-lws-operator/openshift-lws-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.877303857Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-dqnrn/openshift-lws-operator/openshift-lws-operator/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.877359548Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-dqnrn/openshift-lws-operator/openshift-lws-operator/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.877679886Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-dqnrn/openshift-lws-operator/openshift-lws-operator/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.877757318Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-dqnrn/openshift-lws-operator/openshift-lws-operator/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.877803609Z namespaces/openshift-lws-operator/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.87784729Z namespaces/openshift-lws-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.877958032Z namespaces/openshift-lws-operator/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.877992683Z namespaces/openshift-lws-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.878071845Z namespaces/openshift-machine-api/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.878085195Z namespaces/openshift-machine-api/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.878091285Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.878126306Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.878338531Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.878525965Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.878696289Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.878924395Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.87914825Z namespaces/openshift-machine-config-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.87916045Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.87916585Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.879205231Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-controller.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.879332874Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-daemon.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.879425976Z namespaces/openshift-machine-config-operator/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.879436907Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.879468357Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.879655032Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.879845426Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.88002884Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.880278196Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.880478011Z namespaces/openshift-marketplace/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.880493521Z namespaces/openshift-marketplace/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.880500381Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.880515512Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/marketplace-alert-rules.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.880590433Z namespaces/openshift-marketplace/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.880605634Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.880618014Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/certified-operators.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.880737726Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/community-operators.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.880838919Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-marketplace.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.880960052Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-operators.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.881000443Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.881049264Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.881247868Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.881437803Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.881684939Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.881911284Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.882113998Z namespaces/openshift-monitoring/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.882145009Z namespaces/openshift-monitoring/openshift-monitoring.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.882214951Z namespaces/openshift-monitoring/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.882247921Z namespaces/openshift-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.882322023Z namespaces/openshift-monitoring/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.882350304Z namespaces/openshift-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.882492177Z namespaces/openshift-monitoring/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.882957908Z namespaces/openshift-monitoring/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.883312266Z namespaces/openshift-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.883586932Z namespaces/openshift-monitoring/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.883617613Z namespaces/openshift-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.883668734Z namespaces/openshift-monitoring/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.883708826Z namespaces/openshift-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.883787707Z namespaces/openshift-monitoring/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.883880269Z namespaces/openshift-monitoring/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.883941251Z namespaces/openshift-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.884025653Z namespaces/openshift-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.884081634Z namespaces/openshift-monitoring/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.884109035Z namespaces/openshift-monitoring/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.887430981Z namespaces/openshift-monitoring/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.887642377Z namespaces/openshift-monitoring/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.888711381Z namespaces/openshift-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.888774203Z namespaces/openshift-monitoring/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.889796296Z namespaces/openshift-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.88993758Z namespaces/openshift-monitoring/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.890443101Z namespaces/openshift-monitoring/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.890620395Z namespaces/openshift-monitoring/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.890671817Z namespaces/openshift-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.89081601Z namespaces/openshift-monitoring/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.890847791Z namespaces/openshift-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.890936053Z namespaces/openshift-monitoring/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.890983944Z namespaces/openshift-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.891061316Z namespaces/openshift-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.891145618Z namespaces/openshift-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.891191539Z namespaces/openshift-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.89123771Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.891361793Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.891390953Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/main.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.891480175Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.891510416Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/k8s.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.891638249Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.89167056Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/alertmanager-main-rules.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.891780822Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/cluster-monitoring-operator-prometheus-rules.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.891975567Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kube-state-metrics-rules.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.892059549Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kubernetes-monitoring-rules.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.892323785Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/node-exporter-rules.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.89254216Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-prometheus-rules.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.892669973Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-thanos-sidecar-rules.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.892756485Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-operator-rules.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.892883268Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/telemetry.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.89297594Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/thanos-querier.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.893051222Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.893094703Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/alertmanager-main.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.893185145Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/cluster-monitoring-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.893265137Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kube-state-metrics.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.893350229Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kubelet.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.893455091Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/metrics-server.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.893542053Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/node-exporter.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.893720207Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/openshift-state-metrics.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.89381522Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-k8s.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.893931642Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.894019214Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/telemeter-client.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.894125737Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-querier.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.894212109Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-sidecar.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.89427969Z namespaces/openshift-monitoring/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.894333111Z namespaces/openshift-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.894375212Z namespaces/openshift-monitoring/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.894395053Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.894429704Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.894650539Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.894897874Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.895063878Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.895266273Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.895498958Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.895524889Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/openshift-cluster-monitoring.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.895615971Z namespaces/openshift-monitoring/pods/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.895628251Z namespaces/openshift-monitoring/pods/alertmanager-main-0/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.895691683Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager-main-0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.895877257Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.895899528Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.895906048Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.895947359Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.896038781Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.896108833Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.896142203Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.896148274Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.896152203Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.896186724Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.896290937Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.896353308Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.896396699Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.896405129Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.896409509Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.896512132Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.896595114Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.896662555Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.896705926Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.896713747Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.896717707Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.896769278Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.896911301Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.896976753Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.897018963Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.897026724Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.897031074Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.897073385Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.897156057Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.897222008Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.897257099Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.897269339Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.89727554Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.897329771Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.897427043Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.897490614Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.897532266Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.897539936Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.897543916Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.897582487Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.897674609Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.897758321Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.897800432Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-dzpt4/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.897848413Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-dzpt4/cluster-monitoring-operator-548b489cb8-dzpt4.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.897978336Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-dzpt4/cluster-monitoring-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.897995596Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-dzpt4/cluster-monitoring-operator/cluster-monitoring-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.898002086Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-dzpt4/cluster-monitoring-operator/cluster-monitoring-operator/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.898011057Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-dzpt4/cluster-monitoring-operator/cluster-monitoring-operator/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.898997599Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-dzpt4/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.899066281Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-dzpt4/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.899106902Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-2ltcp/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.899142593Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-2ltcp/kube-state-metrics-56d9f944b4-2ltcp.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.899252045Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-2ltcp/kube-rbac-proxy-main/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.899264475Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-2ltcp/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.899272716Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-2ltcp/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.899315817Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-2ltcp/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.899411459Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-2ltcp/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.89947088Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-2ltcp/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.899490071Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-2ltcp/kube-rbac-proxy-self/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.899501501Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-2ltcp/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.899518001Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-2ltcp/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.899537482Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-2ltcp/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.899628224Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-2ltcp/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.899692985Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-2ltcp/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.899733906Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-2ltcp/kube-state-metrics/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.899742957Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-2ltcp/kube-state-metrics/kube-state-metrics/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.899749447Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-2ltcp/kube-state-metrics/kube-state-metrics/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.899780638Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-2ltcp/kube-state-metrics/kube-state-metrics/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.900033933Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-2ltcp/kube-state-metrics/kube-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.900091855Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-2ltcp/kube-state-metrics/kube-state-metrics/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.900125366Z namespaces/openshift-monitoring/pods/metrics-server-67d88cc47d-fdgh8/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.900169187Z namespaces/openshift-monitoring/pods/metrics-server-67d88cc47d-fdgh8/metrics-server-67d88cc47d-fdgh8.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.900265169Z namespaces/openshift-monitoring/pods/metrics-server-67d88cc47d-fdgh8/metrics-server/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.900273669Z namespaces/openshift-monitoring/pods/metrics-server-67d88cc47d-fdgh8/metrics-server/metrics-server/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.900288039Z namespaces/openshift-monitoring/pods/metrics-server-67d88cc47d-fdgh8/metrics-server/metrics-server/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.90032423Z namespaces/openshift-monitoring/pods/metrics-server-67d88cc47d-fdgh8/metrics-server/metrics-server/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.900426013Z namespaces/openshift-monitoring/pods/metrics-server-67d88cc47d-fdgh8/metrics-server/metrics-server/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.900491284Z namespaces/openshift-monitoring/pods/metrics-server-67d88cc47d-fdgh8/metrics-server/metrics-server/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.900530085Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-st8bz/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.900566166Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-st8bz/monitoring-plugin-68b7fc9574-st8bz.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.900661558Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-st8bz/monitoring-plugin/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.900673938Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-st8bz/monitoring-plugin/monitoring-plugin/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.900679138Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-st8bz/monitoring-plugin/monitoring-plugin/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.90077775Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-st8bz/monitoring-plugin/monitoring-plugin/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.900892993Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-st8bz/monitoring-plugin/monitoring-plugin/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.900970035Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-st8bz/monitoring-plugin/monitoring-plugin/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.901016416Z namespaces/openshift-monitoring/pods/node-exporter-7m5zw/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.901094238Z namespaces/openshift-monitoring/pods/node-exporter-7m5zw/node-exporter-7m5zw.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.90119774Z namespaces/openshift-monitoring/pods/node-exporter-7m5zw/init-textfile/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.901206291Z namespaces/openshift-monitoring/pods/node-exporter-7m5zw/init-textfile/init-textfile/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.901210461Z namespaces/openshift-monitoring/pods/node-exporter-7m5zw/init-textfile/init-textfile/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.901254652Z namespaces/openshift-monitoring/pods/node-exporter-7m5zw/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.901320583Z namespaces/openshift-monitoring/pods/node-exporter-7m5zw/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.901387105Z namespaces/openshift-monitoring/pods/node-exporter-7m5zw/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.901438346Z namespaces/openshift-monitoring/pods/node-exporter-7m5zw/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.901446646Z namespaces/openshift-monitoring/pods/node-exporter-7m5zw/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.901456426Z namespaces/openshift-monitoring/pods/node-exporter-7m5zw/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.901528458Z namespaces/openshift-monitoring/pods/node-exporter-7m5zw/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.901642591Z namespaces/openshift-monitoring/pods/node-exporter-7m5zw/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.901707342Z namespaces/openshift-monitoring/pods/node-exporter-7m5zw/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.901757413Z namespaces/openshift-monitoring/pods/node-exporter-7m5zw/node-exporter/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.901765633Z namespaces/openshift-monitoring/pods/node-exporter-7m5zw/node-exporter/node-exporter/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.901771834Z namespaces/openshift-monitoring/pods/node-exporter-7m5zw/node-exporter/node-exporter/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.901811094Z namespaces/openshift-monitoring/pods/node-exporter-7m5zw/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.901972858Z namespaces/openshift-monitoring/pods/node-exporter-7m5zw/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.90205007Z namespaces/openshift-monitoring/pods/node-exporter-7m5zw/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.902119502Z namespaces/openshift-monitoring/pods/node-exporter-jbxlh/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.902174803Z namespaces/openshift-monitoring/pods/node-exporter-jbxlh/node-exporter-jbxlh.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.902296276Z namespaces/openshift-monitoring/pods/node-exporter-jbxlh/init-textfile/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.902313406Z namespaces/openshift-monitoring/pods/node-exporter-jbxlh/init-textfile/init-textfile/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.902321406Z namespaces/openshift-monitoring/pods/node-exporter-jbxlh/init-textfile/init-textfile/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.902360517Z namespaces/openshift-monitoring/pods/node-exporter-jbxlh/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.902447749Z namespaces/openshift-monitoring/pods/node-exporter-jbxlh/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.902539231Z namespaces/openshift-monitoring/pods/node-exporter-jbxlh/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.902585932Z namespaces/openshift-monitoring/pods/node-exporter-jbxlh/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.902597623Z namespaces/openshift-monitoring/pods/node-exporter-jbxlh/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.902621463Z namespaces/openshift-monitoring/pods/node-exporter-jbxlh/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.902662894Z namespaces/openshift-monitoring/pods/node-exporter-jbxlh/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.902769847Z namespaces/openshift-monitoring/pods/node-exporter-jbxlh/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.902851489Z namespaces/openshift-monitoring/pods/node-exporter-jbxlh/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.90293602Z namespaces/openshift-monitoring/pods/node-exporter-jbxlh/node-exporter/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.902950841Z namespaces/openshift-monitoring/pods/node-exporter-jbxlh/node-exporter/node-exporter/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.902961811Z namespaces/openshift-monitoring/pods/node-exporter-jbxlh/node-exporter/node-exporter/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.902996512Z namespaces/openshift-monitoring/pods/node-exporter-jbxlh/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.903128625Z namespaces/openshift-monitoring/pods/node-exporter-jbxlh/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.903207857Z namespaces/openshift-monitoring/pods/node-exporter-jbxlh/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.903260868Z namespaces/openshift-monitoring/pods/node-exporter-ngjnn/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.903311169Z namespaces/openshift-monitoring/pods/node-exporter-ngjnn/node-exporter-ngjnn.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.903418912Z namespaces/openshift-monitoring/pods/node-exporter-ngjnn/init-textfile/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.903427772Z namespaces/openshift-monitoring/pods/node-exporter-ngjnn/init-textfile/init-textfile/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.903434182Z namespaces/openshift-monitoring/pods/node-exporter-ngjnn/init-textfile/init-textfile/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.903482783Z namespaces/openshift-monitoring/pods/node-exporter-ngjnn/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.903569345Z namespaces/openshift-monitoring/pods/node-exporter-ngjnn/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.903652427Z namespaces/openshift-monitoring/pods/node-exporter-ngjnn/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.903700778Z namespaces/openshift-monitoring/pods/node-exporter-ngjnn/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.903714619Z namespaces/openshift-monitoring/pods/node-exporter-ngjnn/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.903724439Z namespaces/openshift-monitoring/pods/node-exporter-ngjnn/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.90375275Z namespaces/openshift-monitoring/pods/node-exporter-ngjnn/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.903884482Z namespaces/openshift-monitoring/pods/node-exporter-ngjnn/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.903973255Z namespaces/openshift-monitoring/pods/node-exporter-ngjnn/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.904020656Z namespaces/openshift-monitoring/pods/node-exporter-ngjnn/node-exporter/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.904028876Z namespaces/openshift-monitoring/pods/node-exporter-ngjnn/node-exporter/node-exporter/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.904038746Z namespaces/openshift-monitoring/pods/node-exporter-ngjnn/node-exporter/node-exporter/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.904095778Z namespaces/openshift-monitoring/pods/node-exporter-ngjnn/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.90420166Z namespaces/openshift-monitoring/pods/node-exporter-ngjnn/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.904291712Z namespaces/openshift-monitoring/pods/node-exporter-ngjnn/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.904323073Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-dgxj6/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.904397874Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-dgxj6/openshift-state-metrics-b8f49f5c6-dgxj6.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.904550508Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-dgxj6/kube-rbac-proxy-main/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.904564268Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-dgxj6/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.904573309Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-dgxj6/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.904608879Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-dgxj6/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.904716182Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-dgxj6/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.904798194Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-dgxj6/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.904842335Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-dgxj6/kube-rbac-proxy-self/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.904850295Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-dgxj6/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.904870765Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-dgxj6/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.904951617Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-dgxj6/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.90504788Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-dgxj6/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.905141972Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-dgxj6/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.905178232Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-dgxj6/openshift-state-metrics/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.905188863Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-dgxj6/openshift-state-metrics/openshift-state-metrics/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.905215923Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-dgxj6/openshift-state-metrics/openshift-state-metrics/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.905272955Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-dgxj6/openshift-state-metrics/openshift-state-metrics/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.905400318Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-dgxj6/openshift-state-metrics/openshift-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.905476679Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-dgxj6/openshift-state-metrics/openshift-state-metrics/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.905555721Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.905642023Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus-k8s-0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.905874999Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.905889829Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.905903219Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.905973901Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.906099154Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.906191746Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.906237257Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.906245307Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.906251367Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.906305949Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.906410931Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.906490813Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.906537484Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.906548764Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.906554924Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.906586655Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.906710838Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.90679416Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.906834131Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.906843021Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.906847041Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.906957654Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.907061416Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.907158378Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.907200659Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.90721246Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.9072184Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.907261681Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.907362483Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.907440775Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.907487486Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.907497686Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.907502116Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.907561448Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.908574611Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.908653383Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.908693244Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.908712164Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.908720014Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.908768676Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.908910529Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.908999571Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.909057282Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-p4prc/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.909104323Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-p4prc/prometheus-operator-84d69f794d-p4prc.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.909226196Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-p4prc/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.909234566Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-p4prc/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.909238586Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-p4prc/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.909285908Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-p4prc/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.909434811Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-p4prc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.909523203Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-p4prc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.909560484Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-p4prc/prometheus-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.909568074Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-p4prc/prometheus-operator/prometheus-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.909577064Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-p4prc/prometheus-operator/prometheus-operator/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.909638896Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-p4prc/prometheus-operator/prometheus-operator/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.909968363Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-p4prc/prometheus-operator/prometheus-operator/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.910054296Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-p4prc/prometheus-operator/prometheus-operator/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.910079106Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-mgf8g/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.910141758Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-mgf8g/prometheus-operator-admission-webhook-7f8d78f54b-mgf8g.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.910302121Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-mgf8g/prometheus-operator-admission-webhook/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.910310561Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-mgf8g/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.910315862Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-mgf8g/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.910372943Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-mgf8g/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.910494275Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-mgf8g/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.910576237Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-mgf8g/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.910611458Z namespaces/openshift-monitoring/pods/telemeter-client-56ff85d9f4-q8crh/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.91070192Z namespaces/openshift-monitoring/pods/telemeter-client-56ff85d9f4-q8crh/telemeter-client-56ff85d9f4-q8crh.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.910902525Z namespaces/openshift-monitoring/pods/telemeter-client-56ff85d9f4-q8crh/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.910915825Z namespaces/openshift-monitoring/pods/telemeter-client-56ff85d9f4-q8crh/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.910925356Z namespaces/openshift-monitoring/pods/telemeter-client-56ff85d9f4-q8crh/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.910991037Z namespaces/openshift-monitoring/pods/telemeter-client-56ff85d9f4-q8crh/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.911095709Z namespaces/openshift-monitoring/pods/telemeter-client-56ff85d9f4-q8crh/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.911169571Z namespaces/openshift-monitoring/pods/telemeter-client-56ff85d9f4-q8crh/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.911210382Z namespaces/openshift-monitoring/pods/telemeter-client-56ff85d9f4-q8crh/reload/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.911218212Z namespaces/openshift-monitoring/pods/telemeter-client-56ff85d9f4-q8crh/reload/reload/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.911224132Z namespaces/openshift-monitoring/pods/telemeter-client-56ff85d9f4-q8crh/reload/reload/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.911273174Z namespaces/openshift-monitoring/pods/telemeter-client-56ff85d9f4-q8crh/reload/reload/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.911384956Z namespaces/openshift-monitoring/pods/telemeter-client-56ff85d9f4-q8crh/reload/reload/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.911466528Z namespaces/openshift-monitoring/pods/telemeter-client-56ff85d9f4-q8crh/reload/reload/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.9115348Z namespaces/openshift-monitoring/pods/telemeter-client-56ff85d9f4-q8crh/telemeter-client/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.91154452Z namespaces/openshift-monitoring/pods/telemeter-client-56ff85d9f4-q8crh/telemeter-client/telemeter-client/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.91154879Z namespaces/openshift-monitoring/pods/telemeter-client-56ff85d9f4-q8crh/telemeter-client/telemeter-client/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.911602231Z namespaces/openshift-monitoring/pods/telemeter-client-56ff85d9f4-q8crh/telemeter-client/telemeter-client/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.911779605Z namespaces/openshift-monitoring/pods/telemeter-client-56ff85d9f4-q8crh/telemeter-client/telemeter-client/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.911853627Z namespaces/openshift-monitoring/pods/telemeter-client-56ff85d9f4-q8crh/telemeter-client/telemeter-client/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.911926019Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.912005761Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/thanos-querier-7dc46f647f-mr6gc.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.912161454Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/kube-rbac-proxy-metrics/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.912174514Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.912179035Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.912214735Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.912317298Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.91240167Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.912503012Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/kube-rbac-proxy-rules/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.912511162Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/kube-rbac-proxy-rules/kube-rbac-proxy-rules/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.912514982Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.912564074Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.912676636Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.912760738Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.912797909Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/kube-rbac-proxy-web/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.912808039Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.912812049Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.912875501Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.913016384Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.913085286Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.913135377Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.913144867Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.913157687Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.913232159Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.913340321Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.913417803Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.913455044Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/prom-label-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.913462684Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.913468595Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.913511045Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.913618338Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.91371296Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.913750921Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/thanos-query/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.913761661Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/thanos-query/thanos-query/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.913769731Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/thanos-query/thanos-query/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.913943266Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/thanos-query/thanos-query/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.914081119Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/thanos-query/thanos-query/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.91415457Z namespaces/openshift-monitoring/pods/thanos-querier-7dc46f647f-mr6gc/thanos-query/thanos-query/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.914206401Z namespaces/openshift-monitoring/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.914249293Z namespaces/openshift-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.914307224Z namespaces/openshift-monitoring/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.914366995Z namespaces/openshift-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.914493138Z namespaces/openshift-multus/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.914539169Z namespaces/openshift-multus/openshift-multus.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.914616501Z namespaces/openshift-multus/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.914674212Z namespaces/openshift-multus/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.914743724Z namespaces/openshift-multus/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.914788755Z namespaces/openshift-multus/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.915161834Z namespaces/openshift-multus/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.915247076Z namespaces/openshift-multus/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.915344478Z namespaces/openshift-multus/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.915397759Z namespaces/openshift-multus/apps/daemonsets/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.91545285Z namespaces/openshift-multus/apps/daemonsets/multus-additional-cni-plugins.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.915607254Z namespaces/openshift-multus/apps/daemonsets/multus.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.915750057Z namespaces/openshift-multus/apps/daemonsets/network-metrics-daemon.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.915827169Z namespaces/openshift-multus/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.91588646Z namespaces/openshift-multus/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.915953532Z namespaces/openshift-multus/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.915991083Z namespaces/openshift-multus/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.916100696Z namespaces/openshift-multus/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.916150297Z namespaces/openshift-multus/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.916181507Z namespaces/openshift-multus/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.91628498Z namespaces/openshift-multus/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.916355461Z namespaces/openshift-multus/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.916404472Z namespaces/openshift-multus/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.916566406Z namespaces/openshift-multus/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.916669149Z namespaces/openshift-multus/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.917414186Z namespaces/openshift-multus/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.917538629Z namespaces/openshift-multus/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.918266526Z namespaces/openshift-multus/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.918372858Z namespaces/openshift-multus/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.918549352Z namespaces/openshift-multus/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.918617804Z namespaces/openshift-multus/core/configmaps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.918736006Z namespaces/openshift-multus/core/configmaps/cni-copy-resources.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.918846709Z namespaces/openshift-multus/core/configmaps/default-cni-sysctl-allowlist.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.918979432Z namespaces/openshift-multus/core/configmaps/multus-daemon-config.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.919079134Z namespaces/openshift-multus/core/configmaps/whereabouts-flatfile-config.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.919188797Z namespaces/openshift-multus/core/serviceaccounts/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.919225768Z namespaces/openshift-multus/core/serviceaccounts/metrics-daemon-sa.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.91933414Z namespaces/openshift-multus/core/serviceaccounts/multus-ac.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.919437923Z namespaces/openshift-multus/core/serviceaccounts/multus-ancillary-tools.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.919542185Z namespaces/openshift-multus/core/serviceaccounts/multus.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.919600066Z namespaces/openshift-multus/core/services/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.919644318Z namespaces/openshift-multus/core/services/network-metrics-service.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.919714959Z namespaces/openshift-multus/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.91975108Z namespaces/openshift-multus/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.919827232Z namespaces/openshift-multus/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.919897213Z namespaces/openshift-multus/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.919967685Z namespaces/openshift-multus/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.920022856Z namespaces/openshift-multus/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.920106778Z namespaces/openshift-multus/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.92019967Z namespaces/openshift-multus/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.920260812Z namespaces/openshift-multus/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.920320433Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.920372434Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.920429156Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/monitor-network.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.920479697Z namespaces/openshift-multus/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.920537678Z namespaces/openshift-multus/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.920587669Z namespaces/openshift-multus/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.9205986Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.920647251Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.920833645Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.92106319Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.921195733Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.921390218Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.921579392Z namespaces/openshift-multus/pods/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.921591983Z namespaces/openshift-multus/pods/multus-5k9gw/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.921616383Z namespaces/openshift-multus/pods/multus-5k9gw/multus-5k9gw.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.921716855Z namespaces/openshift-multus/pods/multus-5k9gw/kube-multus/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.921725266Z namespaces/openshift-multus/pods/multus-5k9gw/kube-multus/kube-multus/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.921731346Z namespaces/openshift-multus/pods/multus-5k9gw/kube-multus/kube-multus/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.921773177Z namespaces/openshift-multus/pods/multus-5k9gw/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.922652917Z namespaces/openshift-multus/pods/multus-5k9gw/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.922728689Z namespaces/openshift-multus/pods/multus-5k9gw/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.922757499Z namespaces/openshift-multus/pods/multus-7h9zp/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.922796911Z namespaces/openshift-multus/pods/multus-7h9zp/multus-7h9zp.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.922997255Z namespaces/openshift-multus/pods/multus-7h9zp/kube-multus/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.923019966Z namespaces/openshift-multus/pods/multus-7h9zp/kube-multus/kube-multus/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.923024876Z namespaces/openshift-multus/pods/multus-7h9zp/kube-multus/kube-multus/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.923058946Z namespaces/openshift-multus/pods/multus-7h9zp/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.923599919Z namespaces/openshift-multus/pods/multus-7h9zp/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.92365275Z namespaces/openshift-multus/pods/multus-7h9zp/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.923685661Z namespaces/openshift-multus/pods/multus-8m5b9/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.923739752Z namespaces/openshift-multus/pods/multus-8m5b9/multus-8m5b9.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.923823064Z namespaces/openshift-multus/pods/multus-8m5b9/kube-multus/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.923831475Z namespaces/openshift-multus/pods/multus-8m5b9/kube-multus/kube-multus/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.923835664Z namespaces/openshift-multus/pods/multus-8m5b9/kube-multus/kube-multus/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.923880376Z namespaces/openshift-multus/pods/multus-8m5b9/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.924737565Z namespaces/openshift-multus/pods/multus-8m5b9/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.924800767Z namespaces/openshift-multus/pods/multus-8m5b9/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.924883709Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.92496027Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/multus-additional-cni-plugins-2mjl8.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.925112924Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/bond-cni-plugin/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.925129105Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.925135654Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.925146075Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.925284448Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.92538075Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.925422151Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/cni-plugins/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.925438832Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/cni-plugins/cni-plugins/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.925442872Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.925490343Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.925585155Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.925650117Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.925691387Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/egress-router-binary-copy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.925700438Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.925704488Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.925740869Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.925853111Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.925970334Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.926005665Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.926016835Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.926029355Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.926083476Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.926167518Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.92625238Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.926291651Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/routeoverride-cni/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.926299692Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.926308572Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.926372303Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.926479086Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.926559927Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.926600288Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/whereabouts-cni-bincopy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.926614799Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.926618839Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.92667012Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.926776243Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.926875495Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.926918846Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/whereabouts-cni/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.926927956Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.926938316Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.926971757Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.92708151Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.927159861Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2mjl8/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.927198792Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.927265654Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/multus-additional-cni-plugins-52wdr.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.927407957Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/bond-cni-plugin/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.927418258Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.927422207Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.927464819Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.927576781Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.927606482Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.927666633Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/cni-plugins/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.927679644Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/cni-plugins/cni-plugins/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.927685544Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.927694714Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.927802056Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.927870708Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.927926149Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/egress-router-binary-copy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.92793965Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.92795803Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.9279772Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.928081073Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.928162805Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.928203166Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.928222936Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.928232826Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.928268707Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.92838249Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.928449021Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.928479422Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/routeoverride-cni/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.928487382Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.928491382Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.928536763Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.928641146Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.928722498Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.928773629Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/whereabouts-cni-bincopy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.928785469Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.92883861Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.928902702Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.929003054Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.929068446Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.929121957Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/whereabouts-cni/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.929130057Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.929133907Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.929186308Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.929295731Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.929375663Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-52wdr/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.929406844Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.929454705Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/multus-additional-cni-plugins-vlzgx.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.929586758Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/bond-cni-plugin/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.929595258Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.929599118Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.929640599Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.929744011Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.929823773Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.929879104Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/cni-plugins/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.929923885Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/cni-plugins/cni-plugins/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.929930426Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.929942116Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.930067279Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.930160141Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.930189192Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/egress-router-binary-copy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.930199612Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.930210772Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.930256263Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.930365706Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.930441797Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.930480688Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.930488709Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.930492759Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.9305508Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.930618912Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.930701634Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.930741864Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/routeoverride-cni/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.930750064Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.930753955Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.930782455Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.93097439Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.931066942Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.931091843Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/whereabouts-cni-bincopy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.931099973Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.931110193Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.931180394Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.931354859Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.931440941Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.931497632Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/whereabouts-cni/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.931510992Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.931516932Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.931562743Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.931667086Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.931742848Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vlzgx/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.931781048Z namespaces/openshift-multus/pods/network-metrics-daemon-d72jm/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.93184653Z namespaces/openshift-multus/pods/network-metrics-daemon-d72jm/network-metrics-daemon-d72jm.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.931981273Z namespaces/openshift-multus/pods/network-metrics-daemon-d72jm/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.931998204Z namespaces/openshift-multus/pods/network-metrics-daemon-d72jm/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.932004234Z namespaces/openshift-multus/pods/network-metrics-daemon-d72jm/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.932019934Z namespaces/openshift-multus/pods/network-metrics-daemon-d72jm/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.932154537Z namespaces/openshift-multus/pods/network-metrics-daemon-d72jm/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.932247289Z namespaces/openshift-multus/pods/network-metrics-daemon-d72jm/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.93227011Z namespaces/openshift-multus/pods/network-metrics-daemon-d72jm/network-metrics-daemon/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.93227652Z namespaces/openshift-multus/pods/network-metrics-daemon-d72jm/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.93228527Z namespaces/openshift-multus/pods/network-metrics-daemon-d72jm/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.932364412Z namespaces/openshift-multus/pods/network-metrics-daemon-d72jm/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.932567627Z namespaces/openshift-multus/pods/network-metrics-daemon-d72jm/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.932658349Z namespaces/openshift-multus/pods/network-metrics-daemon-d72jm/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.93270672Z namespaces/openshift-multus/pods/network-metrics-daemon-s5bhd/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.932762591Z namespaces/openshift-multus/pods/network-metrics-daemon-s5bhd/network-metrics-daemon-s5bhd.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.932883394Z namespaces/openshift-multus/pods/network-metrics-daemon-s5bhd/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.932906314Z namespaces/openshift-multus/pods/network-metrics-daemon-s5bhd/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.932917985Z namespaces/openshift-multus/pods/network-metrics-daemon-s5bhd/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.932927585Z namespaces/openshift-multus/pods/network-metrics-daemon-s5bhd/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.933002657Z namespaces/openshift-multus/pods/network-metrics-daemon-s5bhd/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.933072718Z namespaces/openshift-multus/pods/network-metrics-daemon-s5bhd/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.933113449Z namespaces/openshift-multus/pods/network-metrics-daemon-s5bhd/network-metrics-daemon/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.933122039Z namespaces/openshift-multus/pods/network-metrics-daemon-s5bhd/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.93312606Z namespaces/openshift-multus/pods/network-metrics-daemon-s5bhd/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.933172971Z namespaces/openshift-multus/pods/network-metrics-daemon-s5bhd/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.933365595Z namespaces/openshift-multus/pods/network-metrics-daemon-s5bhd/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.933438537Z namespaces/openshift-multus/pods/network-metrics-daemon-s5bhd/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.933479848Z namespaces/openshift-multus/pods/network-metrics-daemon-v6bp7/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.933537639Z namespaces/openshift-multus/pods/network-metrics-daemon-v6bp7/network-metrics-daemon-v6bp7.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.933624341Z namespaces/openshift-multus/pods/network-metrics-daemon-v6bp7/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.933632961Z namespaces/openshift-multus/pods/network-metrics-daemon-v6bp7/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.933636831Z namespaces/openshift-multus/pods/network-metrics-daemon-v6bp7/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.933683943Z namespaces/openshift-multus/pods/network-metrics-daemon-v6bp7/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.933787625Z namespaces/openshift-multus/pods/network-metrics-daemon-v6bp7/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.933876077Z namespaces/openshift-multus/pods/network-metrics-daemon-v6bp7/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.933934748Z namespaces/openshift-multus/pods/network-metrics-daemon-v6bp7/network-metrics-daemon/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.933943258Z namespaces/openshift-multus/pods/network-metrics-daemon-v6bp7/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.933949189Z namespaces/openshift-multus/pods/network-metrics-daemon-v6bp7/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.93400171Z namespaces/openshift-multus/pods/network-metrics-daemon-v6bp7/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.934187974Z namespaces/openshift-multus/pods/network-metrics-daemon-v6bp7/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.934277496Z namespaces/openshift-multus/pods/network-metrics-daemon-v6bp7/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.934316917Z namespaces/openshift-multus/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.934372008Z namespaces/openshift-multus/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.93442752Z namespaces/openshift-multus/rbac.authorization.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.93443605Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.934484431Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/multus-whereabouts.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.934591093Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.934660395Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.934706786Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.934813009Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/whereabouts-cni.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.93487522Z namespaces/openshift-multus/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.934928761Z namespaces/openshift-multus/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.934986693Z namespaces/openshift-must-gather-q5775/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.934994673Z namespaces/openshift-must-gather-q5775/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.935000683Z namespaces/openshift-must-gather-q5775/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.935061084Z namespaces/openshift-must-gather-q5775/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.935248009Z namespaces/openshift-must-gather-q5775/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.935465294Z namespaces/openshift-must-gather-q5775/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.935631998Z namespaces/openshift-must-gather-q5775/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.936331704Z namespaces/openshift-must-gather-q5775/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.936646701Z namespaces/openshift-network-console/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.937557332Z namespaces/openshift-network-console/openshift-network-console.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.937617524Z namespaces/openshift-network-console/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.937696785Z namespaces/openshift-network-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.937774597Z namespaces/openshift-network-console/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.937819778Z namespaces/openshift-network-console/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.937963522Z namespaces/openshift-network-console/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.938132515Z namespaces/openshift-network-console/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.938278479Z namespaces/openshift-network-console/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.938346171Z namespaces/openshift-network-console/apps/deployments/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.938423162Z namespaces/openshift-network-console/apps/deployments/networking-console-plugin.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.938518584Z namespaces/openshift-network-console/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.938551005Z namespaces/openshift-network-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.938606377Z namespaces/openshift-network-console/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.938678718Z namespaces/openshift-network-console/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.938792361Z namespaces/openshift-network-console/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.938849802Z namespaces/openshift-network-console/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.938898443Z namespaces/openshift-network-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.939040867Z namespaces/openshift-network-console/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.939092198Z namespaces/openshift-network-console/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.939131129Z namespaces/openshift-network-console/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.939267342Z namespaces/openshift-network-console/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.939385064Z namespaces/openshift-network-console/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.939503347Z namespaces/openshift-network-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.939594789Z namespaces/openshift-network-console/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.939738433Z namespaces/openshift-network-console/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.939961748Z namespaces/openshift-network-console/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.940102321Z namespaces/openshift-network-console/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.940174423Z namespaces/openshift-network-console/core/configmaps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.940230284Z namespaces/openshift-network-console/core/configmaps/networking-console-plugin.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.940324486Z namespaces/openshift-network-console/core/services/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.940369137Z namespaces/openshift-network-console/core/services/networking-console-plugin.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.940456869Z namespaces/openshift-network-console/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.94050183Z namespaces/openshift-network-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.940585212Z namespaces/openshift-network-console/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.940630903Z namespaces/openshift-network-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.940700795Z namespaces/openshift-network-console/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.940739046Z namespaces/openshift-network-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.940837548Z namespaces/openshift-network-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.940937131Z namespaces/openshift-network-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.94136561Z namespaces/openshift-network-console/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.941394131Z namespaces/openshift-network-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.941450592Z namespaces/openshift-network-console/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.941508534Z namespaces/openshift-network-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.941573275Z namespaces/openshift-network-console/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.941588295Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.941619926Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.94177762Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.941998145Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.942144499Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.942381574Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.942570998Z namespaces/openshift-network-console/pods/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.942582489Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-nw4hs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.942624459Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-nw4hs/networking-console-plugin-74c9f884b7-nw4hs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.942724252Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-nw4hs/networking-console-plugin/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.942735572Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-nw4hs/networking-console-plugin/networking-console-plugin/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.942740272Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-nw4hs/networking-console-plugin/networking-console-plugin/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.942779183Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-nw4hs/networking-console-plugin/networking-console-plugin/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.942916706Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-nw4hs/networking-console-plugin/networking-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.943006848Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-nw4hs/networking-console-plugin/networking-console-plugin/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.943032609Z namespaces/openshift-network-console/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.94308227Z namespaces/openshift-network-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.943142062Z namespaces/openshift-network-console/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.943188983Z namespaces/openshift-network-console/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.943257414Z namespaces/openshift-network-diagnostics/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.943299385Z namespaces/openshift-network-diagnostics/openshift-network-diagnostics.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.943369317Z namespaces/openshift-network-diagnostics/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.943423118Z namespaces/openshift-network-diagnostics/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.943479429Z namespaces/openshift-network-diagnostics/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.94353052Z namespaces/openshift-network-diagnostics/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.943646913Z namespaces/openshift-network-diagnostics/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.943758136Z namespaces/openshift-network-diagnostics/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.943895059Z namespaces/openshift-network-diagnostics/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.94395171Z namespaces/openshift-network-diagnostics/apps/daemonsets/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.944010982Z namespaces/openshift-network-diagnostics/apps/daemonsets/network-check-target.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.944085623Z namespaces/openshift-network-diagnostics/apps/deployments/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.944136065Z namespaces/openshift-network-diagnostics/apps/deployments/network-check-source.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.944211396Z namespaces/openshift-network-diagnostics/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.944258247Z namespaces/openshift-network-diagnostics/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.944321129Z namespaces/openshift-network-diagnostics/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.94436345Z namespaces/openshift-network-diagnostics/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.944460552Z namespaces/openshift-network-diagnostics/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.944522783Z namespaces/openshift-network-diagnostics/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.944577575Z namespaces/openshift-network-diagnostics/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.944668047Z namespaces/openshift-network-diagnostics/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.944722758Z namespaces/openshift-network-diagnostics/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.944779029Z namespaces/openshift-network-diagnostics/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.944899332Z namespaces/openshift-network-diagnostics/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.945012785Z namespaces/openshift-network-diagnostics/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.94524309Z namespaces/openshift-network-diagnostics/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.945347583Z namespaces/openshift-network-diagnostics/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.945576448Z namespaces/openshift-network-diagnostics/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.94567525Z namespaces/openshift-network-diagnostics/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.945790093Z namespaces/openshift-network-diagnostics/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.945884515Z namespaces/openshift-network-diagnostics/core/serviceaccounts/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.945937366Z namespaces/openshift-network-diagnostics/core/serviceaccounts/network-diagnostics.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.946001918Z namespaces/openshift-network-diagnostics/core/services/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.946050129Z namespaces/openshift-network-diagnostics/core/services/network-check-source.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.946151481Z namespaces/openshift-network-diagnostics/core/services/network-check-target.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.946216463Z namespaces/openshift-network-diagnostics/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.946270394Z namespaces/openshift-network-diagnostics/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.946339885Z namespaces/openshift-network-diagnostics/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.946389457Z namespaces/openshift-network-diagnostics/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.946448188Z namespaces/openshift-network-diagnostics/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.946494609Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.946587241Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.946690254Z namespaces/openshift-network-diagnostics/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.946744525Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.946795176Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.946871898Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.946929299Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/network-check-source.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.946991091Z namespaces/openshift-network-diagnostics/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.947037512Z namespaces/openshift-network-diagnostics/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.947097213Z namespaces/openshift-network-diagnostics/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.947107563Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.947157764Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.947313568Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.947499123Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.94780203Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.948046885Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.94824757Z namespaces/openshift-network-diagnostics/pods/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.9482634Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-888fz/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.948294121Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-888fz/network-check-source-7d88cb55d8-888fz.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.948386983Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-888fz/check-endpoints/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.948394863Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-888fz/check-endpoints/check-endpoints/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.948398853Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-888fz/check-endpoints/check-endpoints/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.948482375Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-888fz/check-endpoints/check-endpoints/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.948600498Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-888fz/check-endpoints/check-endpoints/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.94867743Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-888fz/check-endpoints/check-endpoints/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.948732111Z namespaces/openshift-network-diagnostics/pods/network-check-target-6pt78/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.948787282Z namespaces/openshift-network-diagnostics/pods/network-check-target-6pt78/network-check-target-6pt78.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.948884644Z namespaces/openshift-network-diagnostics/pods/network-check-target-6pt78/network-check-target-container/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.948900725Z namespaces/openshift-network-diagnostics/pods/network-check-target-6pt78/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.948907315Z namespaces/openshift-network-diagnostics/pods/network-check-target-6pt78/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.948946646Z namespaces/openshift-network-diagnostics/pods/network-check-target-6pt78/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.949049238Z namespaces/openshift-network-diagnostics/pods/network-check-target-6pt78/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.9491351Z namespaces/openshift-network-diagnostics/pods/network-check-target-6pt78/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.949179031Z namespaces/openshift-network-diagnostics/pods/network-check-target-q46rj/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.949230863Z namespaces/openshift-network-diagnostics/pods/network-check-target-q46rj/network-check-target-q46rj.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.949309174Z namespaces/openshift-network-diagnostics/pods/network-check-target-q46rj/network-check-target-container/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.949317045Z namespaces/openshift-network-diagnostics/pods/network-check-target-q46rj/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.949320985Z namespaces/openshift-network-diagnostics/pods/network-check-target-q46rj/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.949370936Z namespaces/openshift-network-diagnostics/pods/network-check-target-q46rj/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.949467678Z namespaces/openshift-network-diagnostics/pods/network-check-target-q46rj/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.94954709Z namespaces/openshift-network-diagnostics/pods/network-check-target-q46rj/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.949594941Z namespaces/openshift-network-diagnostics/pods/network-check-target-x5xw7/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.949647262Z namespaces/openshift-network-diagnostics/pods/network-check-target-x5xw7/network-check-target-x5xw7.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.949722724Z namespaces/openshift-network-diagnostics/pods/network-check-target-x5xw7/network-check-target-container/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.949730464Z namespaces/openshift-network-diagnostics/pods/network-check-target-x5xw7/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.949734324Z namespaces/openshift-network-diagnostics/pods/network-check-target-x5xw7/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.949780225Z namespaces/openshift-network-diagnostics/pods/network-check-target-x5xw7/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.949928799Z namespaces/openshift-network-diagnostics/pods/network-check-target-x5xw7/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.95000909Z namespaces/openshift-network-diagnostics/pods/network-check-target-x5xw7/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.950058832Z namespaces/openshift-network-diagnostics/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.950110813Z namespaces/openshift-network-diagnostics/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.950171174Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.950193285Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.950202535Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.950295377Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.950344058Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.950399719Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/network-diagnostics.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.950496092Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.950550333Z namespaces/openshift-network-diagnostics/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.950602914Z namespaces/openshift-network-diagnostics/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.950678356Z namespaces/openshift-network-node-identity/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.950728847Z namespaces/openshift-network-node-identity/openshift-network-node-identity.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.950814099Z namespaces/openshift-network-node-identity/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.950882611Z namespaces/openshift-network-node-identity/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.950944062Z namespaces/openshift-network-node-identity/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.950999573Z namespaces/openshift-network-node-identity/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.951098326Z namespaces/openshift-network-node-identity/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.951196108Z namespaces/openshift-network-node-identity/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.95128755Z namespaces/openshift-network-node-identity/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.951342281Z namespaces/openshift-network-node-identity/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.951399173Z namespaces/openshift-network-node-identity/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.951452704Z namespaces/openshift-network-node-identity/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.951510545Z namespaces/openshift-network-node-identity/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.951611908Z namespaces/openshift-network-node-identity/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.951666519Z namespaces/openshift-network-node-identity/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.95171903Z namespaces/openshift-network-node-identity/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.951832683Z namespaces/openshift-network-node-identity/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.951903405Z namespaces/openshift-network-node-identity/coordination.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.951918475Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.951952235Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ovnkube-identity.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.952025257Z namespaces/openshift-network-node-identity/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.952060158Z namespaces/openshift-network-node-identity/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.95216688Z namespaces/openshift-network-node-identity/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.952259943Z namespaces/openshift-network-node-identity/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.952361545Z namespaces/openshift-network-node-identity/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.952453907Z namespaces/openshift-network-node-identity/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.952552329Z namespaces/openshift-network-node-identity/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.952710763Z namespaces/openshift-network-node-identity/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.952829966Z namespaces/openshift-network-node-identity/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.952911988Z namespaces/openshift-network-node-identity/core/serviceaccounts/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.952959009Z namespaces/openshift-network-node-identity/core/serviceaccounts/network-node-identity.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.953030511Z namespaces/openshift-network-node-identity/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.953075472Z namespaces/openshift-network-node-identity/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.953140853Z namespaces/openshift-network-node-identity/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.953187984Z namespaces/openshift-network-node-identity/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.953254196Z namespaces/openshift-network-node-identity/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.953302127Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.953401629Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.953498111Z namespaces/openshift-network-node-identity/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.953550492Z namespaces/openshift-network-node-identity/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.953603824Z namespaces/openshift-network-node-identity/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.953667925Z namespaces/openshift-network-node-identity/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.953713096Z namespaces/openshift-network-node-identity/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.953778678Z namespaces/openshift-network-node-identity/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.953789808Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.953839429Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.954046824Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.954207548Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.954348731Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.954552286Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.95474732Z namespaces/openshift-network-node-identity/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.954787461Z namespaces/openshift-network-node-identity/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.954873743Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.954896064Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.954950335Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/network-node-identity-leases.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.955001316Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.955054767Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/network-node-identity-leases.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.955114509Z namespaces/openshift-network-node-identity/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.95516718Z namespaces/openshift-network-node-identity/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.955223551Z namespaces/openshift-network-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.955288663Z namespaces/openshift-network-operator/openshift-network-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.955358144Z namespaces/openshift-network-operator/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.955412066Z namespaces/openshift-network-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.955468787Z namespaces/openshift-network-operator/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.955520118Z namespaces/openshift-network-operator/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.955631261Z namespaces/openshift-network-operator/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.955726543Z namespaces/openshift-network-operator/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.955817635Z namespaces/openshift-network-operator/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.955899327Z namespaces/openshift-network-operator/apps/daemonsets/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.955953068Z namespaces/openshift-network-operator/apps/daemonsets/iptables-alerter.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.95602866Z namespaces/openshift-network-operator/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.956078761Z namespaces/openshift-network-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.956141563Z namespaces/openshift-network-operator/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.956189234Z namespaces/openshift-network-operator/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.956283476Z namespaces/openshift-network-operator/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.956340407Z namespaces/openshift-network-operator/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.956406129Z namespaces/openshift-network-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.956490961Z namespaces/openshift-network-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.956547582Z namespaces/openshift-network-operator/coordination.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.956555142Z namespaces/openshift-network-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.956605793Z namespaces/openshift-network-operator/coordination.k8s.io/leases/network-operator-lock.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.956666215Z namespaces/openshift-network-operator/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.956717156Z namespaces/openshift-network-operator/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.957024943Z namespaces/openshift-network-operator/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.957127145Z namespaces/openshift-network-operator/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.957290839Z namespaces/openshift-network-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.957365891Z namespaces/openshift-network-operator/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.957578206Z namespaces/openshift-network-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.957674068Z namespaces/openshift-network-operator/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.957881993Z namespaces/openshift-network-operator/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.957942034Z namespaces/openshift-network-operator/core/configmaps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.957996055Z namespaces/openshift-network-operator/core/configmaps/applied-cluster.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.958093298Z namespaces/openshift-network-operator/core/configmaps/iptables-alerter-script.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.958163879Z namespaces/openshift-network-operator/core/serviceaccounts/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.958221961Z namespaces/openshift-network-operator/core/serviceaccounts/iptables-alerter.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.958295152Z namespaces/openshift-network-operator/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.958333403Z namespaces/openshift-network-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.958400485Z namespaces/openshift-network-operator/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.958448336Z namespaces/openshift-network-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.958508807Z namespaces/openshift-network-operator/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.958557139Z namespaces/openshift-network-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.958664301Z namespaces/openshift-network-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.958757013Z namespaces/openshift-network-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.958829815Z namespaces/openshift-network-operator/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.958901967Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.958971288Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.959029869Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/openshift-network-operator-ipsec-rules.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.959097681Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.959189533Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/network-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.959255175Z namespaces/openshift-network-operator/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.959306666Z namespaces/openshift-network-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.959368837Z namespaces/openshift-network-operator/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.959376798Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.959421088Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.959578332Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.959759646Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.95993329Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.960131335Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.960328229Z namespaces/openshift-network-operator/pods/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.96034032Z namespaces/openshift-network-operator/pods/iptables-alerter-4ksfn/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.960382311Z namespaces/openshift-network-operator/pods/iptables-alerter-4ksfn/iptables-alerter-4ksfn.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.960467633Z namespaces/openshift-network-operator/pods/iptables-alerter-4ksfn/iptables-alerter/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.960475493Z namespaces/openshift-network-operator/pods/iptables-alerter-4ksfn/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.960479393Z namespaces/openshift-network-operator/pods/iptables-alerter-4ksfn/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.960540944Z namespaces/openshift-network-operator/pods/iptables-alerter-4ksfn/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.960634176Z namespaces/openshift-network-operator/pods/iptables-alerter-4ksfn/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.960707128Z namespaces/openshift-network-operator/pods/iptables-alerter-4ksfn/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.960747129Z namespaces/openshift-network-operator/pods/iptables-alerter-9sh7d/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.9608002Z namespaces/openshift-network-operator/pods/iptables-alerter-9sh7d/iptables-alerter-9sh7d.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.960911833Z namespaces/openshift-network-operator/pods/iptables-alerter-9sh7d/iptables-alerter/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.960926313Z namespaces/openshift-network-operator/pods/iptables-alerter-9sh7d/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.960930473Z namespaces/openshift-network-operator/pods/iptables-alerter-9sh7d/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.960937294Z namespaces/openshift-network-operator/pods/iptables-alerter-9sh7d/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.961029526Z namespaces/openshift-network-operator/pods/iptables-alerter-9sh7d/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.961110718Z namespaces/openshift-network-operator/pods/iptables-alerter-9sh7d/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.961157839Z namespaces/openshift-network-operator/pods/iptables-alerter-v9npj/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.96121067Z namespaces/openshift-network-operator/pods/iptables-alerter-v9npj/iptables-alerter-v9npj.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.961291712Z namespaces/openshift-network-operator/pods/iptables-alerter-v9npj/iptables-alerter/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.961299902Z namespaces/openshift-network-operator/pods/iptables-alerter-v9npj/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.961303892Z namespaces/openshift-network-operator/pods/iptables-alerter-v9npj/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.961356723Z namespaces/openshift-network-operator/pods/iptables-alerter-v9npj/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.961453426Z namespaces/openshift-network-operator/pods/iptables-alerter-v9npj/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.961520547Z namespaces/openshift-network-operator/pods/iptables-alerter-v9npj/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.961574518Z namespaces/openshift-network-operator/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.961698461Z namespaces/openshift-network-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.961755153Z namespaces/openshift-network-operator/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.961824344Z namespaces/openshift-network-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.961979668Z namespaces/openshift-node/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.961996878Z namespaces/openshift-node/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.962003218Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.962033049Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.962198033Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.962383707Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.962529511Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.962744225Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.962961541Z namespaces/openshift-operator-lifecycle-manager/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.963013102Z namespaces/openshift-operator-lifecycle-manager/openshift-operator-lifecycle-manager.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.963090684Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.963132164Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.963205316Z namespaces/openshift-operator-lifecycle-manager/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.963243597Z namespaces/openshift-operator-lifecycle-manager/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.96335185Z namespaces/openshift-operator-lifecycle-manager/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.963445422Z namespaces/openshift-operator-lifecycle-manager/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.963540484Z namespaces/openshift-operator-lifecycle-manager/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.963597115Z namespaces/openshift-operator-lifecycle-manager/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.963657197Z namespaces/openshift-operator-lifecycle-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.963711458Z namespaces/openshift-operator-lifecycle-manager/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.963771809Z namespaces/openshift-operator-lifecycle-manager/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.963882682Z namespaces/openshift-operator-lifecycle-manager/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.963949563Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.963999925Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.964104017Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.964161698Z namespaces/openshift-operator-lifecycle-manager/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.96422011Z namespaces/openshift-operator-lifecycle-manager/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.964344073Z namespaces/openshift-operator-lifecycle-manager/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.964453555Z namespaces/openshift-operator-lifecycle-manager/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.964565938Z namespaces/openshift-operator-lifecycle-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.96467244Z namespaces/openshift-operator-lifecycle-manager/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.964759222Z namespaces/openshift-operator-lifecycle-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.964918796Z namespaces/openshift-operator-lifecycle-manager/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.965143071Z namespaces/openshift-operator-lifecycle-manager/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.965207473Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.965248123Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.965301195Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.965349266Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.965428147Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.965475649Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.965590871Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.965691194Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.965739075Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.965781976Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.965825107Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.965898369Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/olm-alert-rules.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.96598078Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.966018241Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.966107343Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.966119154Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.966158384Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.966318418Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.966510223Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.966661656Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.966895981Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.967129527Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.967170938Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/olm-operators.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.96724392Z namespaces/openshift-operator-lifecycle-manager/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.967302341Z namespaces/openshift-operator-lifecycle-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.967369623Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.967429044Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.967532076Z namespaces/openshift-operators/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.967542007Z namespaces/openshift-operators/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.967546077Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.967549887Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.967733281Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.967982487Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.968151071Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.968368456Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.968672463Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.968700653Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/global-operators.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.968787265Z namespaces/openshift-ovn-kubernetes/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.968846407Z namespaces/openshift-ovn-kubernetes/openshift-ovn-kubernetes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.968955299Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.96898521Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.969062122Z namespaces/openshift-ovn-kubernetes/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.969082532Z namespaces/openshift-ovn-kubernetes/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.969274177Z namespaces/openshift-ovn-kubernetes/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.969346638Z namespaces/openshift-ovn-kubernetes/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.96942102Z namespaces/openshift-ovn-kubernetes/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.969478241Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.969500582Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ovnkube-node.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.969640155Z namespaces/openshift-ovn-kubernetes/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.969673256Z namespaces/openshift-ovn-kubernetes/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.969710137Z namespaces/openshift-ovn-kubernetes/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.969754448Z namespaces/openshift-ovn-kubernetes/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.969828449Z namespaces/openshift-ovn-kubernetes/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.969901771Z namespaces/openshift-ovn-kubernetes/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.969931202Z namespaces/openshift-ovn-kubernetes/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.970007404Z namespaces/openshift-ovn-kubernetes/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.970066115Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.970077345Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.970087915Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ovn-kubernetes-master.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.970157557Z namespaces/openshift-ovn-kubernetes/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.970178937Z namespaces/openshift-ovn-kubernetes/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.970434514Z namespaces/openshift-ovn-kubernetes/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.970536566Z namespaces/openshift-ovn-kubernetes/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.970882654Z namespaces/openshift-ovn-kubernetes/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.971016567Z namespaces/openshift-ovn-kubernetes/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.971499548Z namespaces/openshift-ovn-kubernetes/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.97160091Z namespaces/openshift-ovn-kubernetes/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.971768914Z namespaces/openshift-ovn-kubernetes/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.971832216Z namespaces/openshift-ovn-kubernetes/core/configmaps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.971900428Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-config.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.9720313Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-script-lib.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.972155503Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.972198064Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-control-plane.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.972327247Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-node.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.972401649Z namespaces/openshift-ovn-kubernetes/core/services/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.97244185Z namespaces/openshift-ovn-kubernetes/core/services/ovn-kubernetes-node.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.972520942Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.972557373Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.972655175Z namespaces/openshift-ovn-kubernetes/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.972691126Z namespaces/openshift-ovn-kubernetes/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.972763767Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.972775258Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.972817479Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/default.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.972903431Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.972948022Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.973073864Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.973167097Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.973226728Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.97328338Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.973353361Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.973383662Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/networking-rules.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.973480304Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.973508665Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/monitor-ovn-node.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.973569776Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.973580776Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.973624657Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ovn.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.9737221Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/signer.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.973779511Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.973825952Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.973902684Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.973918274Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.973953805Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.974212611Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.974402675Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.974545379Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.974762914Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.97501943Z namespaces/openshift-ovn-kubernetes/pods/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.97503498Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.975096471Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/ovnkube-node-bd48h.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.975281496Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/kube-rbac-proxy-node/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.975293856Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.975299916Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.975308846Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.975445459Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.975509351Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.975549152Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.975559542Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.975563672Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.975578903Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.975738196Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.975819758Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.975850449Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/nbdb/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.975870689Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/nbdb/nbdb/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.97587774Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/nbdb/nbdb/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.975935001Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/nbdb/nbdb/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.976046213Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.976141166Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.976168336Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/northd/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.976172906Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/northd/northd/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.976176827Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/northd/northd/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.976216787Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/northd/northd/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.97634886Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.976427102Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/northd/northd/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.976467763Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/ovn-acl-logging/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.976478193Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.976484443Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.976527095Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.976622527Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.976684518Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.976726689Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/ovn-controller/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.9767447Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/ovn-controller/ovn-controller/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.9767492Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.976798251Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.977113018Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.97719436Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.977237041Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/ovnkube-controller/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.977251211Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.977258072Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.977306913Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.979024152Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.979077074Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.979107944Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/sbdb/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.979117835Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/sbdb/sbdb/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.979123695Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/sbdb/sbdb/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.979170056Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/sbdb/sbdb/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.979282738Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.97936273Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-bd48h/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.979397401Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.979488583Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/ovnkube-node-grkdn.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.979639347Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/kube-rbac-proxy-node/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.979650877Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.979666337Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.979734229Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.979886182Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.979976444Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.980004715Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.980009365Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.980013445Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.980052196Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.980198229Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.980277021Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.980307032Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/nbdb/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.980320632Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/nbdb/nbdb/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.980326863Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/nbdb/nbdb/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.980385584Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/nbdb/nbdb/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.980497147Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.980574118Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.980608909Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/northd/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.980619529Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/northd/northd/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.98063361Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/northd/northd/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.980678131Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/northd/northd/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.980788283Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.980871245Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/northd/northd/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.980909056Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/ovn-acl-logging/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.980924296Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.980929437Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.980985098Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.98109875Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.981176152Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.981217943Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/ovn-controller/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.981228753Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/ovn-controller/ovn-controller/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.981234533Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.981289995Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.981575061Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.981667514Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.981713825Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/ovnkube-controller/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.981727255Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.981732525Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.981764826Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.983410864Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.983487356Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.983528857Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/sbdb/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.983542467Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/sbdb/sbdb/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.983546487Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/sbdb/sbdb/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.983590518Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/sbdb/sbdb/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.983719551Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.983797353Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-grkdn/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.983839524Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.983947826Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/ovnkube-node-m9k9v.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.98410381Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/kube-rbac-proxy-node/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.98411636Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.9841224Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.984153051Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.984308265Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.984373516Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.984410057Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.984432358Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.984437188Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.984443468Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.984565711Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.984634622Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.984669573Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/nbdb/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.984676943Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/nbdb/nbdb/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.984680793Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/nbdb/nbdb/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.984709424Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/nbdb/nbdb/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.984803616Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.984886768Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.984938649Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/northd/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.98495205Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/northd/northd/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.98496319Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/northd/northd/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.984990331Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/northd/northd/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.985110413Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.985176735Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/northd/northd/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.985206876Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/ovn-acl-logging/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.985214216Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.985227286Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.985258147Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.985341509Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.98538973Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/ovn-controller/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.98539762Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/ovn-controller/ovn-controller/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.98540172Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.985442791Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.98582322Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.985923222Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.985961073Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/ovnkube-controller/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.985968683Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.985974513Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.986030615Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.987777335Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.987833606Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.987875987Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/sbdb/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.987900548Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/sbdb/sbdb/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.987908968Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/sbdb/sbdb/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.987944399Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/sbdb/sbdb/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.988057191Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.988138564Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-m9k9v/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.988175134Z namespaces/openshift-ovn-kubernetes/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.988235496Z namespaces/openshift-ovn-kubernetes/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.988296327Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.988304437Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.988354278Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.988459121Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-nodes-identity-limited.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.988595904Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.988656925Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.988707446Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.988811309Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.988927492Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.988982593Z namespaces/openshift-ovn-kubernetes/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.989026014Z namespaces/openshift-ovn-kubernetes/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.989094006Z namespaces/openshift-route-controller-manager/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.989102046Z namespaces/openshift-route-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.989106036Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.989155997Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/openshift-route-controllers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.989213208Z namespaces/openshift-route-controller-manager/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.989221138Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.989345761Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.989563136Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.989787942Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.989957886Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.99016138Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.990389565Z namespaces/openshift-service-ca-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.990447357Z namespaces/openshift-service-ca-operator/openshift-service-ca-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.990513478Z namespaces/openshift-service-ca-operator/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.990546649Z namespaces/openshift-service-ca-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.99059883Z namespaces/openshift-service-ca-operator/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.990650822Z namespaces/openshift-service-ca-operator/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.990750024Z namespaces/openshift-service-ca-operator/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.990870447Z namespaces/openshift-service-ca-operator/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.990967969Z namespaces/openshift-service-ca-operator/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.99102946Z namespaces/openshift-service-ca-operator/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.991059371Z namespaces/openshift-service-ca-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.991119342Z namespaces/openshift-service-ca-operator/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.991147173Z namespaces/openshift-service-ca-operator/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.991236115Z namespaces/openshift-service-ca-operator/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.991269646Z namespaces/openshift-service-ca-operator/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.991317117Z namespaces/openshift-service-ca-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.991394849Z namespaces/openshift-service-ca-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.99143431Z namespaces/openshift-service-ca-operator/coordination.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.99144227Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.991500581Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/service-ca-operator-lock.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.991563553Z namespaces/openshift-service-ca-operator/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.991620474Z namespaces/openshift-service-ca-operator/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.991738947Z namespaces/openshift-service-ca-operator/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.991838249Z namespaces/openshift-service-ca-operator/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.992086515Z namespaces/openshift-service-ca-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.992173477Z namespaces/openshift-service-ca-operator/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.99230268Z namespaces/openshift-service-ca-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.992457183Z namespaces/openshift-service-ca-operator/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.992637797Z namespaces/openshift-service-ca-operator/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.992707939Z namespaces/openshift-service-ca-operator/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.99275724Z namespaces/openshift-service-ca-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.992826512Z namespaces/openshift-service-ca-operator/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.992890463Z namespaces/openshift-service-ca-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.992958275Z namespaces/openshift-service-ca-operator/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.992987306Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.993075498Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.99315076Z namespaces/openshift-service-ca-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.99318982Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.993238922Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.993301213Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.993348674Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/service-ca-operator.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.993418826Z namespaces/openshift-service-ca-operator/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.993460977Z namespaces/openshift-service-ca-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.993530168Z namespaces/openshift-service-ca-operator/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.993538729Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.993582179Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.993736883Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.994070901Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.994218554Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.994419809Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.994604903Z namespaces/openshift-service-ca-operator/pods/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.994613113Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-kj6ht/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.994678745Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-kj6ht/service-ca-operator-5d6c75f84f-kj6ht.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.994762487Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-kj6ht/service-ca-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.994770897Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-kj6ht/service-ca-operator/service-ca-operator/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.994775197Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-kj6ht/service-ca-operator/service-ca-operator/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.994831488Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-kj6ht/service-ca-operator/service-ca-operator/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.995031893Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-kj6ht/service-ca-operator/service-ca-operator/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.995185676Z namespaces/openshift-service-ca-operator/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.995214797Z namespaces/openshift-service-ca-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.995270209Z namespaces/openshift-service-ca-operator/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.99531611Z namespaces/openshift-service-ca-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.995379771Z namespaces/openshift-service-ca/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.995414922Z namespaces/openshift-service-ca/openshift-service-ca.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.995478253Z namespaces/openshift-service-ca/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.995511314Z namespaces/openshift-service-ca/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.995565025Z namespaces/openshift-service-ca/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.995594266Z namespaces/openshift-service-ca/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.995702748Z namespaces/openshift-service-ca/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.995822631Z namespaces/openshift-service-ca/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.995949584Z namespaces/openshift-service-ca/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.995995205Z namespaces/openshift-service-ca/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.996056827Z namespaces/openshift-service-ca/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.996113798Z namespaces/openshift-service-ca/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.996164279Z namespaces/openshift-service-ca/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.996258541Z namespaces/openshift-service-ca/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.996321613Z namespaces/openshift-service-ca/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.996371314Z namespaces/openshift-service-ca/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.996465566Z namespaces/openshift-service-ca/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.996524347Z namespaces/openshift-service-ca/coordination.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.996532488Z namespaces/openshift-service-ca/coordination.k8s.io/leases/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.996581499Z namespaces/openshift-service-ca/coordination.k8s.io/leases/service-ca-controller-lock.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.99664235Z namespaces/openshift-service-ca/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.996695011Z namespaces/openshift-service-ca/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.996809234Z namespaces/openshift-service-ca/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.996917907Z namespaces/openshift-service-ca/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.997040399Z namespaces/openshift-service-ca/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.997131582Z namespaces/openshift-service-ca/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.997253784Z namespaces/openshift-service-ca/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.997393138Z namespaces/openshift-service-ca/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.997573692Z namespaces/openshift-service-ca/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.997643773Z namespaces/openshift-service-ca/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.997743176Z namespaces/openshift-service-ca/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.997793667Z namespaces/openshift-service-ca/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.997824148Z namespaces/openshift-service-ca/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.99790137Z namespaces/openshift-service-ca/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.99793316Z namespaces/openshift-service-ca/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.998020642Z namespaces/openshift-service-ca/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.998099104Z namespaces/openshift-service-ca/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.998145785Z namespaces/openshift-service-ca/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.998176996Z namespaces/openshift-service-ca/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.998227707Z namespaces/openshift-service-ca/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.998260058Z namespaces/openshift-service-ca/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.998310679Z namespaces/openshift-service-ca/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.998319019Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.99836315Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.998513834Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.998707188Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.998851701Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.999090287Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.999272421Z namespaces/openshift-service-ca/pods/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.999286451Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-vn449/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.999318512Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-vn449/service-ca-658d7b7c78-vn449.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.999453385Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-vn449/service-ca-controller/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.999461666Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-vn449/service-ca-controller/service-ca-controller/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.999465656Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-vn449/service-ca-controller/service-ca-controller/logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.999505057Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-vn449/service-ca-controller/service-ca-controller/logs/current.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.999760253Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-vn449/service-ca-controller/service-ca-controller/logs/previous.insecure.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.999827034Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-vn449/service-ca-controller/service-ca-controller/logs/previous.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.999877145Z namespaces/openshift-service-ca/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:37.999992538Z namespaces/openshift-service-ca/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.000052699Z namespaces/openshift-service-ca/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.000111211Z namespaces/openshift-service-ca/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.000152282Z namespaces/openshift-user-workload-monitoring/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.000222443Z namespaces/openshift-user-workload-monitoring/openshift-user-workload-monitoring.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.000302155Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.000364816Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.000426008Z namespaces/openshift-user-workload-monitoring/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.000482809Z namespaces/openshift-user-workload-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.000580061Z namespaces/openshift-user-workload-monitoring/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.000669103Z namespaces/openshift-user-workload-monitoring/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.000764266Z namespaces/openshift-user-workload-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.000805127Z namespaces/openshift-user-workload-monitoring/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.000874488Z namespaces/openshift-user-workload-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.00094471Z namespaces/openshift-user-workload-monitoring/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.000977181Z namespaces/openshift-user-workload-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.001052382Z namespaces/openshift-user-workload-monitoring/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.001094743Z namespaces/openshift-user-workload-monitoring/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.001147185Z namespaces/openshift-user-workload-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.001236867Z namespaces/openshift-user-workload-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.001278848Z namespaces/openshift-user-workload-monitoring/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.001344849Z namespaces/openshift-user-workload-monitoring/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.001455992Z namespaces/openshift-user-workload-monitoring/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.001547704Z namespaces/openshift-user-workload-monitoring/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.001643196Z namespaces/openshift-user-workload-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.001733068Z namespaces/openshift-user-workload-monitoring/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.00182716Z namespaces/openshift-user-workload-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.002002114Z namespaces/openshift-user-workload-monitoring/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.002163188Z namespaces/openshift-user-workload-monitoring/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.002207699Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.00225257Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.002310652Z namespaces/openshift-user-workload-monitoring/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.002360173Z namespaces/openshift-user-workload-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.002421914Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.002457265Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.002564438Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.0026758Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.002712271Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.002775472Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.002837224Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.002907485Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.002980967Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.002996327Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.003032448Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.003184432Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.003359256Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.003502719Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.003697324Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.003909429Z namespaces/openshift-user-workload-monitoring/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.003935449Z namespaces/openshift-user-workload-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.00399116Z namespaces/openshift-user-workload-monitoring/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.004049332Z namespaces/openshift-user-workload-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.004106093Z namespaces/openshift/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.004163204Z namespaces/openshift/openshift.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.004228396Z namespaces/openshift/apps.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.004267777Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.004337539Z namespaces/openshift/apps/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.004376869Z namespaces/openshift/apps/daemonsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.004478812Z namespaces/openshift/apps/deployments.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.004569534Z namespaces/openshift/apps/replicasets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.004666566Z namespaces/openshift/apps/statefulsets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.004737978Z namespaces/openshift/autoscaling/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.004787079Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.00485196Z namespaces/openshift/batch/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.004904441Z namespaces/openshift/batch/cronjobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.005015454Z namespaces/openshift/batch/jobs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.005069195Z namespaces/openshift/build.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.005121447Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.005221259Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.00527596Z namespaces/openshift/core/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.005330051Z namespaces/openshift/core/configmaps.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.005436624Z namespaces/openshift/core/endpoints.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.005527096Z namespaces/openshift/core/events.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.005619888Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.00571508Z namespaces/openshift/core/pods.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.005807153Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.005961116Z namespaces/openshift/core/secrets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.00613479Z namespaces/openshift/core/services.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.006246613Z namespaces/openshift/discovery.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.006294284Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.006358315Z namespaces/openshift/image.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.006396236Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.007752147Z namespaces/openshift/image.openshift.io/imagestreams/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.007776408Z namespaces/openshift/image.openshift.io/imagestreams/cli-artifacts.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.008009164Z namespaces/openshift/image.openshift.io/imagestreams/cli.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.008107126Z namespaces/openshift/image.openshift.io/imagestreams/dotnet-runtime.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.008231239Z namespaces/openshift/image.openshift.io/imagestreams/dotnet.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.008358012Z namespaces/openshift/image.openshift.io/imagestreams/driver-toolkit.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.008466594Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift-java11.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.008634538Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.008806022Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java-openshift.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.008968956Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java11-openshift.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.009085498Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift-jdk11.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.009196981Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.009335534Z namespaces/openshift/image.openshift.io/imagestreams/golang.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.009449437Z namespaces/openshift/image.openshift.io/imagestreams/httpd.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.00956816Z namespaces/openshift/image.openshift.io/imagestreams/installer-artifacts.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.009675632Z namespaces/openshift/image.openshift.io/imagestreams/installer.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.009789075Z namespaces/openshift/image.openshift.io/imagestreams/java-runtime.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.009929438Z namespaces/openshift/image.openshift.io/imagestreams/java.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.010062481Z namespaces/openshift/image.openshift.io/imagestreams/jboss-datagrid73-openshift.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.010198744Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-openshift.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.010315567Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.01044273Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-openshift.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.010558303Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.010800768Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-openshift.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.010922031Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.011046544Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-openshift.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.011164846Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-runtime-openshift.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.011280679Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk11-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.011410862Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk8-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.011539985Z namespaces/openshift/image.openshift.io/imagestreams/jenkins-agent-base.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.011656408Z namespaces/openshift/image.openshift.io/imagestreams/jenkins.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.011785891Z namespaces/openshift/image.openshift.io/imagestreams/mariadb.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.011942865Z namespaces/openshift/image.openshift.io/imagestreams/must-gather.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.012047547Z namespaces/openshift/image.openshift.io/imagestreams/mysql.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.012208891Z namespaces/openshift/image.openshift.io/imagestreams/network-tools.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.012313273Z namespaces/openshift/image.openshift.io/imagestreams/nginx.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.012455607Z namespaces/openshift/image.openshift.io/imagestreams/nodejs.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.01262729Z namespaces/openshift/image.openshift.io/imagestreams/oauth-proxy.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.012732323Z namespaces/openshift/image.openshift.io/imagestreams/openjdk-11-rhel7.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.012890137Z namespaces/openshift/image.openshift.io/imagestreams/perl.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.013023249Z namespaces/openshift/image.openshift.io/imagestreams/php.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.013160823Z namespaces/openshift/image.openshift.io/imagestreams/postgresql.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.013295496Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso75-openshift-rhel8.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.013397698Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso76-openshift-rhel8.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.013503821Z namespaces/openshift/image.openshift.io/imagestreams/python.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.013642864Z namespaces/openshift/image.openshift.io/imagestreams/redhat-openjdk18-openshift.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.013784357Z namespaces/openshift/image.openshift.io/imagestreams/redis.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.01392791Z namespaces/openshift/image.openshift.io/imagestreams/ruby.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.014055693Z namespaces/openshift/image.openshift.io/imagestreams/sso75-openshift-rhel8.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.014164566Z namespaces/openshift/image.openshift.io/imagestreams/sso76-openshift-rhel8.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.014270669Z namespaces/openshift/image.openshift.io/imagestreams/tests.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.014402972Z namespaces/openshift/image.openshift.io/imagestreams/tools.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.014532875Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11-runtime.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.014679618Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.014814191Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17-runtime.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.014978365Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.015114008Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21-runtime.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.015222991Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.015330613Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8-runtime.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.015462316Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.015576249Z namespaces/openshift/k8s.ovn.org/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.01564658Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.015740852Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.015844955Z namespaces/openshift/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.015986488Z namespaces/openshift/monitoring.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.016022209Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.016089481Z namespaces/openshift/networking.k8s.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.016132172Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.016208103Z namespaces/openshift/operators.coreos.com/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.016224164Z namespaces/openshift/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.016252284Z namespaces/openshift/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.016411768Z namespaces/openshift/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.016602312Z namespaces/openshift/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.016747306Z namespaces/openshift/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.016981941Z namespaces/openshift/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.017168905Z namespaces/openshift/policy/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.017212677Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.017283248Z namespaces/openshift/route.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.017322479Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.017392581Z namespaces/openshift/template.openshift.io/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.017401541Z namespaces/openshift/template.openshift.io/templates/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.017443892Z namespaces/openshift/template.openshift.io/templates/cache-service.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.017576055Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-example.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.017708678Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-persistent.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.017830131Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-example.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.017977924Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-persistent.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.018157259Z namespaces/openshift/template.openshift.io/templates/datagrid-service.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.018281871Z namespaces/openshift/template.openshift.io/templates/django-psql-example.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.018406634Z namespaces/openshift/template.openshift.io/templates/django-psql-persistent.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.018539797Z namespaces/openshift/template.openshift.io/templates/eap-xp3-basic-s2i.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.018721732Z namespaces/openshift/template.openshift.io/templates/eap-xp4-basic-s2i.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.018888765Z namespaces/openshift/template.openshift.io/templates/eap74-basic-s2i.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.019012618Z namespaces/openshift/template.openshift.io/templates/eap74-https-s2i.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.019160312Z namespaces/openshift/template.openshift.io/templates/eap74-sso-s2i.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.019316505Z namespaces/openshift/template.openshift.io/templates/httpd-example.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.019449498Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral-monitored.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.019596952Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.019733085Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent-monitored.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.01993745Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.020063213Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.020187735Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.020322129Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.020439401Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.020551954Z namespaces/openshift/template.openshift.io/templates/mariadb-ephemeral.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.020667097Z namespaces/openshift/template.openshift.io/templates/mariadb-persistent.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.020775569Z namespaces/openshift/template.openshift.io/templates/mysql-ephemeral.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.020918632Z namespaces/openshift/template.openshift.io/templates/mysql-persistent.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.021034635Z namespaces/openshift/template.openshift.io/templates/nginx-example.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.021153968Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-example.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.021295171Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-persistent.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.021428304Z namespaces/openshift/template.openshift.io/templates/openjdk-web-basic-s2i.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.021539507Z namespaces/openshift/template.openshift.io/templates/postgresql-ephemeral.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.021647959Z namespaces/openshift/template.openshift.io/templates/postgresql-persistent.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.021769172Z namespaces/openshift/template.openshift.io/templates/rails-pgsql-persistent.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.021932766Z namespaces/openshift/template.openshift.io/templates/rails-postgresql-example.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.022062739Z namespaces/openshift/template.openshift.io/templates/react-web-app-example.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.022167391Z namespaces/openshift/template.openshift.io/templates/redis-ephemeral.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.022283964Z namespaces/openshift/template.openshift.io/templates/redis-persistent.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.022401497Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-rest-3scale.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.02253035Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-xml.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.022666313Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.022769175Z namespaces/openshift/template.openshift.io/templates/sso75-https.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.022924199Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-https.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.023045332Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.023182735Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql-persistent.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.023321488Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.023466411Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-https.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.023926622Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql-persistent.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.024083416Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.024222899Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-https.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.024342832Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.024435384Z network_logs/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.024494045Z network_logs/cluster_scale [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.024580367Z network_logs/ippools.whereabouts.cni.cncf.io [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.024678939Z network_logs/multi-networkpolicy [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.024746971Z network_logs/net-attach-def [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.024871504Z network_logs/overlappingrangeipreservations.whereabouts.cni.cncf.io [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.02512799Z network_logs/ovn_kubernetes_top_pods [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.025234303Z network_logs/ovnk_database_store.tar.gz [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.027580007Z nodes/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.027615498Z nodes/debug [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.02772235Z nodes/ip-10-0-132-194.ec2.internal/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.027756101Z nodes/ip-10-0-132-194.ec2.internal/cpu_affinities.json [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.028062678Z nodes/ip-10-0-132-194.ec2.internal/dmesg [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.028352004Z nodes/ip-10-0-132-194.ec2.internal/ethtool_channels [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.028432297Z nodes/ip-10-0-132-194.ec2.internal/ethtool_features [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.028509438Z nodes/ip-10-0-132-194.ec2.internal/ip-10-0-132-194.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.028972679Z nodes/ip-10-0-132-194.ec2.internal/irq_affinities.json [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.029118482Z nodes/ip-10-0-132-194.ec2.internal/lscpu [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.029243085Z nodes/ip-10-0-132-194.ec2.internal/lspci [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.029338967Z nodes/ip-10-0-132-194.ec2.internal/podresources.json [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.0294402Z nodes/ip-10-0-132-194.ec2.internal/pods_info.json [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.029566103Z nodes/ip-10-0-132-194.ec2.internal/proc_cmdline [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.029633984Z nodes/ip-10-0-132-194.ec2.internal/sysinfo.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.031518558Z nodes/ip-10-0-132-194.ec2.internal/sysinfo.tgz [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.031655891Z nodes/ip-10-0-137-134.ec2.internal/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.031690592Z nodes/ip-10-0-137-134.ec2.internal/cpu_affinities.json [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.03205577Z nodes/ip-10-0-137-134.ec2.internal/dmesg [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.032322387Z nodes/ip-10-0-137-134.ec2.internal/ethtool_channels [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.032402638Z nodes/ip-10-0-137-134.ec2.internal/ethtool_features [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.0324957Z nodes/ip-10-0-137-134.ec2.internal/ip-10-0-137-134.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.03289791Z nodes/ip-10-0-137-134.ec2.internal/irq_affinities.json [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.033074874Z nodes/ip-10-0-137-134.ec2.internal/lscpu [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.033193397Z nodes/ip-10-0-137-134.ec2.internal/lspci [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.033278509Z nodes/ip-10-0-137-134.ec2.internal/podresources.json [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.033389151Z nodes/ip-10-0-137-134.ec2.internal/pods_info.json [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.033502444Z nodes/ip-10-0-137-134.ec2.internal/proc_cmdline [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.033582736Z nodes/ip-10-0-137-134.ec2.internal/sysinfo.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.03551785Z nodes/ip-10-0-137-134.ec2.internal/sysinfo.tgz [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.035676384Z nodes/ip-10-0-143-86.ec2.internal/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.035699635Z nodes/ip-10-0-143-86.ec2.internal/cpu_affinities.json [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.036121795Z nodes/ip-10-0-143-86.ec2.internal/dmesg [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.036393461Z nodes/ip-10-0-143-86.ec2.internal/ethtool_channels [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.036469192Z nodes/ip-10-0-143-86.ec2.internal/ethtool_features [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.036547814Z nodes/ip-10-0-143-86.ec2.internal/ip-10-0-143-86.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.037041556Z nodes/ip-10-0-143-86.ec2.internal/irq_affinities.json [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.037175829Z nodes/ip-10-0-143-86.ec2.internal/lscpu [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.037302542Z nodes/ip-10-0-143-86.ec2.internal/lspci [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.037394454Z nodes/ip-10-0-143-86.ec2.internal/podresources.json [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.037502907Z nodes/ip-10-0-143-86.ec2.internal/pods_info.json [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.037617939Z nodes/ip-10-0-143-86.ec2.internal/proc_cmdline [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.037699621Z nodes/ip-10-0-143-86.ec2.internal/sysinfo.log [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.039631086Z nodes/ip-10-0-143-86.ec2.internal/sysinfo.tgz [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.039795089Z pod_network_connectivity_check/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.03983859Z pod_network_connectivity_check/podnetworkconnectivitychecks.yaml [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.039976824Z static-pods/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.039993824Z static-pods/kube-apiserver/ [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.047936558Z [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.047963719Z sent 72,986 bytes received 8,350,238 bytes 5,615,482.67 bytes/sec [must-gather] [must-gather-dw5cm] OUT 2026-07-27T18:31:38.047974529Z total size is 111,924,940 speedup is 13.29 [must-gather] [must-gather ] OUT 2026-07-27T18:31:38.383452077Z namespace/openshift-must-gather-q5775 deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 67b080d3-17dc-4b35-8a6a-b965aa8fce53 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [git-push-artifacts] WORK_DIR: /workspace/odh-ci-artifacts [git-push-artifacts] REPO_PATH: opendatahub-io/odh-build-metadata [git-push-artifacts] REPO_BRANCH: ci-artifacts [git-push-artifacts] SPARSE_FILE_PATH: test-artifacts/docs [git-push-artifacts] SOURCE_PATH: /workspace/artifacts-dir [git-push-artifacts] DEST_PATH: test-artifacts/kserve-group-test-tqx2c [git-push-artifacts] ALWAYS_PASS: false [git-push-artifacts] configuring gh token [git-push-artifacts] taking github token from Konflux bot [git-push-artifacts] Initialized empty Git repository in /workspace/odh-ci-artifacts/.git/ [git-push-artifacts] Using partial fetch with sparse checkout for: test-artifacts/docs [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] * [new branch] ci-artifacts -> origin/ci-artifacts [git-push-artifacts] Already on 'ci-artifacts' [git-push-artifacts] branch 'ci-artifacts' set up to track 'origin/ci-artifacts'. [git-push-artifacts] TASK_NAME=kserve-group-test-tqx2c-e2e-llm-inference-service [git-push-artifacts] PIPELINERUN_NAME=kserve-group-test-tqx2c [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] -rw-r--r--. 1 root 1001540000 17059114 Jul 27 18:33 /workspace/odh-ci-artifacts/test-artifacts/kserve-group-test-tqx2c/e2e-llm-inference-service.tar.gz [git-push-artifacts] [ci-artifacts 01dea35] Updating CI Artifacts in e2e-llm-inference-service [git-push-artifacts] 1 file changed, 0 insertions(+), 0 deletions(-) [git-push-artifacts] create mode 100644 test-artifacts/kserve-group-test-tqx2c/e2e-llm-inference-service.tar.gz [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] To https://github.com/opendatahub-io/odh-build-metadata.git [git-push-artifacts] c5190ba..01dea35 ci-artifacts -> ci-artifacts [fail-if-needed] Failing pipeline because deploy-and-e2e step failed container step-fail-if-needed has failed : [{"key":"StartedAt","value":"2026-07-27T18:33:29.111Z","type":3}]