task e2e-llm-inference-service has failed: "step-fail-if-needed" exited with code 1: Error [get-kubeconfig] Found kubeconfig secret: cluster-glf5j-admin-kubeconfig [get-kubeconfig] Wrote kubeconfig to /credentials/cluster-glf5j-kubeconfig [get-kubeconfig] Found admin password secret: cluster-glf5j-admin-password [get-kubeconfig] Retrieved username [get-kubeconfig] Wrote password to /credentials/cluster-glf5j-password [get-kubeconfig] API Server URL: https://a3cde18be540944adbcbe5b2ee93881c-9d3d79d2762ce307.elb.us-east-1.amazonaws.com:6443 [get-kubeconfig] Console URL: https://console-openshift-console.apps.1334158e-23ec-4349-bd08-e85981d41f8e.prod.konfluxeaas.com [clone-repo] RHOAIENG-55229/cert-manager-webhook-race [clone-repo] https://github.com/jlost/kserve [clone-repo] Cloning into '/workspace/source'... [clone-repo] Updating files: 85% (2657/3111) Updating files: 86% (2676/3111) Updating files: 87% (2707/3111) Updating files: 88% (2738/3111) Updating files: 89% (2769/3111) Updating files: 90% (2800/3111) Updating files: 91% (2832/3111) Updating files: 92% (2863/3111) Updating files: 93% (2894/3111) Updating files: 94% (2925/3111) Updating files: 95% (2956/3111) Updating files: 96% (2987/3111) Updating files: 97% (3018/3111) Updating files: 98% (3049/3111) Updating files: 99% (3080/3111) Updating files: 100% (3111/3111) Updating files: 100% (3111/3111), done. [e2e-llm-inference-service] + bash [e2e-llm-inference-service] + STATUS_FILE=/test-status/deploy-and-e2e-status [e2e-llm-inference-service] + echo failed [e2e-llm-inference-service] + COMPONENT_NAME=kserve-agent-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-agent-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:4e90f566b342ee88137b4332294c4cef6db04c33dbbb6e9394e58bda8902c06d [e2e-llm-inference-service] + KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:4e90f566b342ee88137b4332294c4cef6db04c33dbbb6e9394e58bda8902c06d [e2e-llm-inference-service] + COMPONENT_NAME=kserve-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:b726734d76566f6525bce03e3e8254c7f011cd4de9eb8c3c32567009a372fe75 [e2e-llm-inference-service] + KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:b726734d76566f6525bce03e3e8254c7f011cd4de9eb8c3c32567009a372fe75 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-router-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-router-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:814af323b5d1cbdab913eb2692456d1382c8eaf87a5271e4958bd36fa46947bd [e2e-llm-inference-service] + KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:814af323b5d1cbdab913eb2692456d1382c8eaf87a5271e4958bd36fa46947bd [e2e-llm-inference-service] + COMPONENT_NAME=kserve-storage-initializer-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-storage-initializer-ci '.[$component_name].image' [e2e-llm-inference-service] + export STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1 [e2e-llm-inference-service] + STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1 [e2e-llm-inference-service] + COMPONENT_NAME=odh-kserve-llmisvc-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name odh-kserve-llmisvc-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:ae1ed5810fd25d49c55490e1c452cc8365e6e9ba8f5c764a4a8d47dc7f0af5c1 [e2e-llm-inference-service] + LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:ae1ed5810fd25d49c55490e1c452cc8365e6e9ba8f5c764a4a8d47dc7f0af5c1 [e2e-llm-inference-service] + ./test/scripts/openshift-ci/run-e2e-tests.sh 'llminferenceservice and cluster_cpu and not autoscaling and not tracing' 2 llm-d [e2e-llm-inference-service] INFERENCE_POOL_GROUP=inference.networking.k8s.io (detected from OCP 4.21.24) [e2e-llm-inference-service] Installing on cluster [e2e-llm-inference-service] Using namespace: kserve for KServe components [e2e-llm-inference-service] SKLEARN_IMAGE=quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1730 [e2e-llm-inference-service] OPT_125M_MODEL_URI=s3://example-models/facebook/opt-125m [e2e-llm-inference-service] ERROR_404_ISVC_IMAGE=quay.io/opendatahub/error-404-isvc:odh-pr-1730 [e2e-llm-inference-service] SUCCESS_200_ISVC_IMAGE=quay.io/opendatahub/success-200-isvc:odh-pr-1730 [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Kustomize v5.8.1 to /workspace/source/bin/kustomize [e2e-llm-inference-service] v5.8.1 [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] [INFO] Installing yq v4.52.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed yq v4.52.1 to /workspace/source/bin/yq [e2e-llm-inference-service] yq (https://github.com/mikefarah/yq/) version v4.52.1 [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Installing KServe Python SDK ... [e2e-llm-inference-service] [INFO] Installing uv 0.7.8 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed uv 0.7.8 to /workspace/source/bin/uv [e2e-llm-inference-service] warning: Failed to read project metadata (No `pyproject.toml` found in current directory or any parent directory). Running `uv self version` for compatibility. This fallback will be removed in the future; pass `--preview` to force an error. [e2e-llm-inference-service] uv 0.7.8 [e2e-llm-inference-service] Creating virtual environment... [e2e-llm-inference-service] warning: virtualenv's `--clear` has no effect (uv always clears the virtual environment) [e2e-llm-inference-service] Using CPython 3.9.25 interpreter at: /usr/bin/python3 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] Using CPython 3.11.13 interpreter at: /usr/bin/python3.11 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] Resolved 266 packages in 1ms [e2e-llm-inference-service] Building kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading cryptography (4.3MiB) [e2e-llm-inference-service] Downloading pandas (12.5MiB) [e2e-llm-inference-service] Downloading pydantic-core (2.0MiB) [e2e-llm-inference-service] Downloading grpcio (6.4MiB) [e2e-llm-inference-service] Downloading aiohttp (1.7MiB) [e2e-llm-inference-service] Downloading pyarrow (40.1MiB) [e2e-llm-inference-service] Downloading grpcio-tools (2.5MiB) [e2e-llm-inference-service] Downloading numpy (15.7MiB) [e2e-llm-inference-service] Downloading black (1.6MiB) [e2e-llm-inference-service] Downloading uvloop (3.8MiB) [e2e-llm-inference-service] Downloading portforward (3.9MiB) [e2e-llm-inference-service] Downloading kubernetes (1.9MiB) [e2e-llm-inference-service] Downloading mypy (17.2MiB) [e2e-llm-inference-service] Downloading setuptools (1.2MiB) [e2e-llm-inference-service] Downloading botocore (12.9MiB) [e2e-llm-inference-service] Building timeout-sampler==1.0.3 [e2e-llm-inference-service] Building python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading aiohttp [e2e-llm-inference-service] Downloading pydantic-core [e2e-llm-inference-service] Downloading black [e2e-llm-inference-service] Downloading grpcio-tools [e2e-llm-inference-service] Downloading setuptools [e2e-llm-inference-service] Downloading portforward [e2e-llm-inference-service] Downloading uvloop [e2e-llm-inference-service] Downloading cryptography [e2e-llm-inference-service] Downloading grpcio [e2e-llm-inference-service] Downloading kubernetes [e2e-llm-inference-service] Built python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading numpy [e2e-llm-inference-service] Downloading pandas [e2e-llm-inference-service] Downloading botocore [e2e-llm-inference-service] Downloading pyarrow [e2e-llm-inference-service] Built timeout-sampler==1.0.3 [e2e-llm-inference-service] Downloading mypy [e2e-llm-inference-service] Built kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Prepared 101 packages in 2.01s [e2e-llm-inference-service] warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance. [e2e-llm-inference-service] If the cache and target directories are on different filesystems, hardlinking may not be supported. [e2e-llm-inference-service] If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning. [e2e-llm-inference-service] Installed 101 packages in 698ms [e2e-llm-inference-service] + aiohappyeyeballs==2.6.1 [e2e-llm-inference-service] + aiohttp==3.13.3 [e2e-llm-inference-service] + aiosignal==1.4.0 [e2e-llm-inference-service] + annotated-doc==0.0.4 [e2e-llm-inference-service] + annotated-types==0.7.0 [e2e-llm-inference-service] + anyio==4.9.0 [e2e-llm-inference-service] + attrs==25.3.0 [e2e-llm-inference-service] + avro==1.12.0 [e2e-llm-inference-service] + black==24.3.0 [e2e-llm-inference-service] + boto3==1.37.35 [e2e-llm-inference-service] + botocore==1.37.35 [e2e-llm-inference-service] + cachetools==5.5.2 [e2e-llm-inference-service] + certifi==2025.1.31 [e2e-llm-inference-service] + cffi==2.0.0 [e2e-llm-inference-service] + charset-normalizer==3.4.1 [e2e-llm-inference-service] + click==8.1.8 [e2e-llm-inference-service] + cloudevents==1.11.0 [e2e-llm-inference-service] + colorama==0.4.6 [e2e-llm-inference-service] + colorlog==6.10.1 [e2e-llm-inference-service] + coverage==7.8.0 [e2e-llm-inference-service] + cryptography==46.0.5 [e2e-llm-inference-service] + deprecation==2.1.0 [e2e-llm-inference-service] + durationpy==0.9 [e2e-llm-inference-service] + execnet==2.1.1 [e2e-llm-inference-service] + fastapi==0.136.3 [e2e-llm-inference-service] + frozenlist==1.5.0 [e2e-llm-inference-service] + google-auth==2.39.0 [e2e-llm-inference-service] + grpc-interceptor==0.15.4 [e2e-llm-inference-service] + grpcio==1.78.1 [e2e-llm-inference-service] + grpcio-testing==1.78.1 [e2e-llm-inference-service] + grpcio-tools==1.78.1 [e2e-llm-inference-service] + h11==0.16.0 [e2e-llm-inference-service] + httpcore==1.0.9 [e2e-llm-inference-service] + httptools==0.6.4 [e2e-llm-inference-service] + httpx==0.27.2 [e2e-llm-inference-service] + httpx-retries==0.4.5 [e2e-llm-inference-service] + idna==3.10 [e2e-llm-inference-service] + iniconfig==2.1.0 [e2e-llm-inference-service] + jinja2==3.1.6 [e2e-llm-inference-service] + jmespath==1.0.1 [e2e-llm-inference-service] + kserve==0.19.0 (from file:///workspace/source/python/kserve) [e2e-llm-inference-service] + kubernetes==32.0.1 [e2e-llm-inference-service] + markupsafe==3.0.2 [e2e-llm-inference-service] + multidict==6.4.3 [e2e-llm-inference-service] + mypy==0.991 [e2e-llm-inference-service] + mypy-extensions==1.0.0 [e2e-llm-inference-service] + numpy==2.2.4 [e2e-llm-inference-service] + oauthlib==3.2.2 [e2e-llm-inference-service] + orjson==3.10.16 [e2e-llm-inference-service] + packaging==24.2 [e2e-llm-inference-service] + pandas==2.2.3 [e2e-llm-inference-service] + pathspec==0.12.1 [e2e-llm-inference-service] + platformdirs==4.3.7 [e2e-llm-inference-service] + pluggy==1.5.0 [e2e-llm-inference-service] + portforward==0.7.1 [e2e-llm-inference-service] + prometheus-client==0.21.1 [e2e-llm-inference-service] + propcache==0.3.1 [e2e-llm-inference-service] + protobuf==6.33.5 [e2e-llm-inference-service] + psutil==5.9.8 [e2e-llm-inference-service] + pyarrow==19.0.1 [e2e-llm-inference-service] + pyasn1==0.6.3 [e2e-llm-inference-service] + pyasn1-modules==0.4.2 [e2e-llm-inference-service] + pycparser==2.22 [e2e-llm-inference-service] + pydantic==2.12.4 [e2e-llm-inference-service] + pydantic-core==2.41.5 [e2e-llm-inference-service] + pyjwt==2.12.1 [e2e-llm-inference-service] + pytest==7.4.4 [e2e-llm-inference-service] + pytest-asyncio==0.23.8 [e2e-llm-inference-service] + pytest-cov==5.0.0 [e2e-llm-inference-service] + pytest-httpx==0.30.0 [e2e-llm-inference-service] + pytest-json-report==1.5.0 [e2e-llm-inference-service] + pytest-metadata==3.1.1 [e2e-llm-inference-service] + pytest-xdist==3.6.1 [e2e-llm-inference-service] + python-dateutil==2.9.0.post0 [e2e-llm-inference-service] + python-dotenv==1.1.0 [e2e-llm-inference-service] + python-multipart==0.0.22 [e2e-llm-inference-service] + python-simple-logger==2.0.19 [e2e-llm-inference-service] + pytz==2025.2 [e2e-llm-inference-service] + pyyaml==6.0.2 [e2e-llm-inference-service] + requests==2.32.3 [e2e-llm-inference-service] + requests-oauthlib==2.0.0 [e2e-llm-inference-service] + rsa==4.9.1 [e2e-llm-inference-service] + s3transfer==0.11.4 [e2e-llm-inference-service] + setuptools==78.1.0 [e2e-llm-inference-service] + six==1.17.0 [e2e-llm-inference-service] + sniffio==1.3.1 [e2e-llm-inference-service] + starlette==1.2.1 [e2e-llm-inference-service] + tabulate==0.9.0 [e2e-llm-inference-service] + timeout-sampler==1.0.3 [e2e-llm-inference-service] + timing-asgi==0.3.1 [e2e-llm-inference-service] + tomlkit==0.13.2 [e2e-llm-inference-service] + typing-extensions==4.15.0 [e2e-llm-inference-service] + typing-inspection==0.4.2 [e2e-llm-inference-service] + tzdata==2025.2 [e2e-llm-inference-service] + urllib3==2.6.2 [e2e-llm-inference-service] + uvicorn==0.34.1 [e2e-llm-inference-service] + uvloop==0.21.0 [e2e-llm-inference-service] + watchfiles==1.0.5 [e2e-llm-inference-service] + websocket-client==1.8.0 [e2e-llm-inference-service] + websockets==15.0.1 [e2e-llm-inference-service] + yarl==1.20.0 [e2e-llm-inference-service] Audited 1 package in 48ms [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Now using project "kserve" on server "https://a3cde18be540944adbcbe5b2ee93881c-9d3d79d2762ce307.elb.us-east-1.amazonaws.com:6443". [e2e-llm-inference-service] [e2e-llm-inference-service] You can add applications to this project with the 'new-app' command. For example, try: [e2e-llm-inference-service] [e2e-llm-inference-service] oc new-app rails-postgresql-example [e2e-llm-inference-service] [e2e-llm-inference-service] to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: [e2e-llm-inference-service] [e2e-llm-inference-service] kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- /agnhost serve-hostname [e2e-llm-inference-service] [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Creating namespace openshift-keda... [e2e-llm-inference-service] namespace/openshift-keda created [e2e-llm-inference-service] Namespace openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating OperatorGroup openshift-keda... [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-keda created [e2e-llm-inference-service] OperatorGroup openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating Subscription for openshift-custom-metrics-autoscaler-operator... [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-custom-metrics-autoscaler-operator created [e2e-llm-inference-service] Subscription openshift-custom-metrics-autoscaler-operator created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Waiting for openshift-custom-metrics-autoscaler-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (20/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (40/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (45/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 is ready (Phase: Succeeded). [e2e-llm-inference-service] --- [e2e-llm-inference-service] Applying KedaController custom resource... [e2e-llm-inference-service] Warning: resource kedacontrollers/keda is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] kedacontroller.keda.sh/keda configured [e2e-llm-inference-service] KedaController custom resource applied. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Allowing time for KEDA components to be provisioned by the operator ... [e2e-llm-inference-service] Waiting for KEDA Operator pod (selector: "app=keda-operator") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-operator" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-operator" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-operator" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-operator-77f648d694-6rspr 1/1 Running 0 41s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-operator" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-operator-77f648d694-6rspr condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-operator" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Operator pod is ready. [e2e-llm-inference-service] Waiting for KEDA Metrics API Server pod (selector: "app=keda-metrics-apiserver") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-metrics-apiserver" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-metrics-apiserver-55557df8c-td5fs 1/1 Running 0 47s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-metrics-apiserver-55557df8c-td5fs condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Metrics API Server pod is ready. [e2e-llm-inference-service] Waiting for KEDA Webhook pod (selector: "app=keda-admission-webhooks") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-admission-webhooks" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-admission-56f9798bb4-fq48f 1/1 Running 0 52s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-admission-56f9798bb4-fq48f condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Webhook pod is ready. [e2e-llm-inference-service] --- [e2e-llm-inference-service] ✅ KEDA deployment script finished successfully. [e2e-llm-inference-service] KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:b726734d76566f6525bce03e3e8254c7f011cd4de9eb8c3c32567009a372fe75 [e2e-llm-inference-service] LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:ae1ed5810fd25d49c55490e1c452cc8365e6e9ba8f5c764a4a8d47dc7f0af5c1 [e2e-llm-inference-service] KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:4e90f566b342ee88137b4332294c4cef6db04c33dbbb6e9394e58bda8902c06d [e2e-llm-inference-service] KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:814af323b5d1cbdab913eb2692456d1382c8eaf87a5271e4958bd36fa46947bd [e2e-llm-inference-service] STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1 [e2e-llm-inference-service] Installing KServe via kustomize... [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepoolimports.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for CRDs to be established... [e2e-llm-inference-service] Waiting for CRD inferenceservices.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD inferenceservices.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD llminferenceserviceconfigs.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD llminferenceserviceconfigs.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD clusterstoragecontainers.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD clusterstoragecontainers.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD datascienceclusters.datasciencecluster.opendatahub.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met [e2e-llm-inference-service] Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepoolimports.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for llmisvc-controller-manager to be ready... [e2e-llm-inference-service] Waiting for pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to be created... [e2e-llm-inference-service] Pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" found. [e2e-llm-inference-service] Current pods for -l "control-plane=llmisvc-controller-manager" in namespace "kserve": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] llmisvc-controller-manager-bdf85754c-vt4bs 0/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to become ready... [e2e-llm-inference-service] pod/llmisvc-controller-manager-bdf85754c-vt4bs condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" are ready. [e2e-llm-inference-service] Re-applying LLMInferenceServiceConfig resources with webhook validation... [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-decode-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-decode-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-prefill-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-prefill-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-scheduler is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] Applying DSC/DSCI resources... [e2e-llm-inference-service] dscinitialization.dscinitialization.opendatahub.io/test-dsci created [e2e-llm-inference-service] datasciencecluster.datasciencecluster.opendatahub.io/test-dsc created [e2e-llm-inference-service] KServe manual installation complete [e2e-llm-inference-service] 🔧 Configuration: [e2e-llm-inference-service] KServe deployment: ❌ disabled [e2e-llm-inference-service] Kuadrant deployment: ✅ enabled [e2e-llm-inference-service] [e2e-llm-inference-service] Checking OpenShift server version...(4.21.24) [e2e-llm-inference-service] 🎯 Server version (4.21.24) is 4.19.9 or higher - continue with the script [e2e-llm-inference-service] ⏳ Installing cert-manager [e2e-llm-inference-service] namespace/cert-manager-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] Waiting for openshift-cert-manager-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (10/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 found, but not yet Succeeded (Phase: Installing). Waiting... (15/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD certificates.cert-manager.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD certificates.cert-manager.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io condition met [e2e-llm-inference-service] ✅ Cert-manager installed [e2e-llm-inference-service] ⏳ Installing openshift-lws-operator [e2e-llm-inference-service] namespace/openshift-lws-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] subscription.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] Waiting for leader-worker-set CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (10/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (15/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD leaderworkersetoperators.operator.openshift.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD leaderworkersetoperators.operator.openshift.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/leaderworkersetoperators.operator.openshift.io condition met [e2e-llm-inference-service] leaderworkersetoperator.operator.openshift.io/cluster created [e2e-llm-inference-service] ⏳ waiting for openshift-lws-operator to be ready.… [e2e-llm-inference-service] Waiting for pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to be created... [e2e-llm-inference-service] Pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" found. [e2e-llm-inference-service] Current pods for -l "name=openshift-lws-operator" in namespace "openshift-lws-operator": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-lws-operator-fd8ccff4c-mrl5g 1/1 Running 0 12s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to become ready... [e2e-llm-inference-service] pod/openshift-lws-operator-fd8ccff4c-mrl5g condition met [e2e-llm-inference-service] Pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" are ready. [e2e-llm-inference-service] ✅ openshift-lws-operator installed [e2e-llm-inference-service] gatewayclass.gateway.networking.k8s.io/openshift-default created [e2e-llm-inference-service] Waiting for pod -l "app=istiod" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "app=istiod" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "app=istiod" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] istiod-openshift-gateway-94bb8fbfd-zprl6 1/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "app=istiod" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/istiod-openshift-gateway-94bb8fbfd-zprl6 condition met [e2e-llm-inference-service] Pod(s) -l "app=istiod" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Creating a Gateway [e2e-llm-inference-service] Error from server (AlreadyExists): namespaces "openshift-ingress" already exists [e2e-llm-inference-service] ⏳ Creating gateway memory ConfigMap for parametersRef (2Gi) [e2e-llm-inference-service] configmap/gateway-proxy-config created [e2e-llm-inference-service] gateway.gateway.networking.k8s.io/openshift-ai-inference created [e2e-llm-inference-service] Waiting for pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-ai-inference-openshift-default-655f7c58b4-8g97n 0/1 ContainerCreating 0 5s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/openshift-ai-inference-openshift-default-655f7c58b4-8g97n condition met [e2e-llm-inference-service] Pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Installing RHCL(Kuadrant) operator [e2e-llm-inference-service] namespace/kuadrant-system created [e2e-llm-inference-service] subscription.operators.coreos.com/rhcl-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/kuadrant created [e2e-llm-inference-service] Waiting for rhcl-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (20/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD kuadrants.kuadrant.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD kuadrants.kuadrant.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/kuadrants.kuadrant.io condition met [e2e-llm-inference-service] Waiting for apiserver discovery /apis/kuadrant.io/v1beta1 to list kuadrants (timeout: 120s)… [e2e-llm-inference-service] Discovery for kuadrant.io/v1beta1 includes kuadrants. [e2e-llm-inference-service] ⏳ sleeping 30s after discovery (RESTMapper can trail discovery)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant created [e2e-llm-inference-service] ⏳ waiting for Kuadrant Ready (attempt 1/2, timeout 5m)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant condition met [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-6d85f6564-xdn96 1/1 Running 0 2m27s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-6d85f6564-xdn96 condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ⏳ waiting for authorino service to be created... [e2e-llm-inference-service] service/authorino-authorino-authorization condition met [e2e-llm-inference-service] service/authorino-authorino-authorization annotated [e2e-llm-inference-service] Warning: resource authorinos/authorino is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] authorino.operator.authorino.kuadrant.io/authorino configured [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-6d85f6564-xdn96 1/1 Running 0 2m37s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-6d85f6564-xdn96 condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ✅ kuadrant(authorino) installed [e2e-llm-inference-service] Patching ingress domain... [e2e-llm-inference-service] configmap/inferenceservice-config patched [e2e-llm-inference-service] pod "kserve-controller-manager-6b69b6d547-f58tf" deleted [e2e-llm-inference-service] Waiting for kserve-controller-manager to be ready... [e2e-llm-inference-service] pod/kserve-controller-manager-6b69b6d547-w9wkj condition met [e2e-llm-inference-service] Installing ODH Model Controller manually... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/accounts.nim.opendatahub.io created [e2e-llm-inference-service] serviceaccount/model-serving-api created [e2e-llm-inference-service] serviceaccount/odh-model-controller created [e2e-llm-inference-service] role.rbac.authorization.k8s.io/leader-election-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-editor-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-viewer-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-prometheus-k8s created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/metrics-reader created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/odh-model-controller-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/proxy-role created [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/odh-model-controller-rolebinding-opendatahub created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/proxy-rolebinding created [e2e-llm-inference-service] configmap/odh-model-controller-parameters created [e2e-llm-inference-service] service/model-serving-api created [e2e-llm-inference-service] service/odh-model-controller-metrics-service created [e2e-llm-inference-service] service/odh-model-controller-webhook-service created [e2e-llm-inference-service] deployment.apps/model-serving-api created [e2e-llm-inference-service] deployment.apps/odh-model-controller created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/model-serving-api-metrics created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/odh-model-controller-metrics-monitor created [e2e-llm-inference-service] template.template.openshift.io/autogluon-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/guardrails-detector-huggingface-serving-template created [e2e-llm-inference-service] template.template.openshift.io/kserve-ovms created [e2e-llm-inference-service] template.template.openshift.io/mlserver-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-2 created [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/validating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] Waiting for deployment "odh-model-controller" rollout to finish: 0 of 1 updated replicas are available... [e2e-llm-inference-service] deployment "odh-model-controller" successfully rolled out [e2e-llm-inference-service] networkpolicy.networking.k8s.io/allow-all created [e2e-llm-inference-service] KServe setup complete (namespace: kserve) [e2e-llm-inference-service] Add testing models to SeaweedFS S3 storage ... [e2e-llm-inference-service] Waiting for SeaweedFS deployment to be ready... [e2e-llm-inference-service] deployment "seaweedfs" successfully rolled out [e2e-llm-inference-service] S3 init job not completed, re-creating... [e2e-llm-inference-service] job.batch/s3-init replaced [e2e-llm-inference-service] Waiting for S3 init job to complete... [e2e-llm-inference-service] job.batch/s3-init condition met [e2e-llm-inference-service] Prepare CI namespace and install ServingRuntimes [e2e-llm-inference-service] Setting up CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Tearing down CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Namespace kserve-ci-e2e-test does not exist, skipping deletion [e2e-llm-inference-service] CI namespace teardown complete [e2e-llm-inference-service] Creating namespace kserve-ci-e2e-test [e2e-llm-inference-service] namespace/kserve-ci-e2e-test created [e2e-llm-inference-service] Applying S3 artifact secret [e2e-llm-inference-service] secret/mlpipeline-s3-artifact created [e2e-llm-inference-service] Applying storage-config secret [e2e-llm-inference-service] secret/storage-config created [e2e-llm-inference-service] Applying SeaweedFS S3 credentials secret [e2e-llm-inference-service] secret/seaweedfs-s3-creds created [e2e-llm-inference-service] Linking seaweedfs-s3-creds to default service account [e2e-llm-inference-service] Creating odh-trusted-ca-bundle configmap [e2e-llm-inference-service] configmap/odh-trusted-ca-bundle created [e2e-llm-inference-service] Installing ServingRuntimes [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-autogluonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver-multinode created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-lgbserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-mlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-paddleserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-pmmlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-predictiveserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-sklearnserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tensorflow-serving created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-torchserve created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tritonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-vllmserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-xgbserver created [e2e-llm-inference-service] CI namespace setup complete [e2e-llm-inference-service] Setup complete [e2e-llm-inference-service] === E2E cluster / operator summary === [e2e-llm-inference-service] Client Version: 4.20.11 [e2e-llm-inference-service] Kustomize Version: v5.6.0 [e2e-llm-inference-service] Server Version: 4.21.24 [e2e-llm-inference-service] Kubernetes Version: v1.34.9 [e2e-llm-inference-service] ClusterVersion desired: 4.21.24 [e2e-llm-inference-service] ClusterVersion history (latest): 4.21.24 (Completed) [e2e-llm-inference-service] CSVs in kuadrant-system: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in openshift-keda: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] custom-metrics-autoscaler.v2.19.0-2 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in cert-manager-operator: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in openshift-lws-operator: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] leader-worker-set.v1.0.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in openshift-operators (ODH / shared operators, filtered): [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] Kuadrant / Authorino (diagnostics): [e2e-llm-inference-service] CRD kuadrants.kuadrant.io versions: v1beta1 served=true storage=true [e2e-llm-inference-service] Subscriptions in kuadrant-system: [e2e-llm-inference-service] authorino-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators authorino-operator.v1.4.1 [e2e-llm-inference-service] dns-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators dns-operator.v1.4.0 [e2e-llm-inference-service] limitador-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators limitador-operator.v1.4.0 [e2e-llm-inference-service] rhcl-operator stable redhat-operators rhcl-operator.v1.4.1 [e2e-llm-inference-service] Kuadrant CR conditions (kuadrant/kuadrant-system): [e2e-llm-inference-service] Ready=True (Ready) [e2e-llm-inference-service] KServe deployments in kserve: [e2e-llm-inference-service] kserve-controller-manager: ready=1 image=quay.io/opendatahub/kserve-controller@sha256:b726734d76566f6525bce03e3e8254c7f011cd4de9eb8c3c32567009a372fe75 [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-controller@sha256:b726734d76566f6525bce03e3e8254c7f011cd4de9eb8c3c32567009a372fe75 [e2e-llm-inference-service] odh-model-controller: ready=1 image=quay.io/opendatahub/odh-model-controller:fast [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-model-controller@sha256:5ce69c3ae5bc6b3b4888343ec0c55c7e1c8a05304b25bfbd157358fc4c26ee52 [e2e-llm-inference-service] llmisvc-controller-manager: ready=1 image=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:ae1ed5810fd25d49c55490e1c452cc8365e6e9ba8f5c764a4a8d47dc7f0af5c1 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:16b0c45ad086e41bcb2ffd306971370cfa25d620863c44826dfef1764a1cc96a [e2e-llm-inference-service] === End E2E cluster / operator summary === [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] CA certificate extracted [e2e-llm-inference-service] REQUESTS_CA_BUNDLE=/tmp/ca.crt [e2e-llm-inference-service] Run E2E tests: llminferenceservice and cluster_cpu and not autoscaling and not tracing [e2e-llm-inference-service] Starting E2E functional tests ... [e2e-llm-inference-service] Parallelism requested for pytest is 2 [e2e-llm-inference-service] ============================= test session starts ============================== [e2e-llm-inference-service] platform linux -- Python 3.11.13, pytest-7.4.4, pluggy-1.5.0 -- /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] cachedir: .pytest_cache [e2e-llm-inference-service] metadata: {'Python': '3.11.13', 'Platform': 'Linux-5.14.0-570.122.1.el9_6.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.5.0'}, 'Plugins': {'json-report': '1.5.0', 'asyncio': '0.23.8', 'anyio': '4.9.0', 'httpx': '0.30.0', 'cov': '5.0.0', 'xdist': '3.6.1', 'metadata': '3.1.1'}, 'PLATFORM': 'el9'} [e2e-llm-inference-service] rootdir: /workspace/source/test/e2e [e2e-llm-inference-service] configfile: pytest.ini [e2e-llm-inference-service] plugins: json-report-1.5.0, asyncio-0.23.8, anyio-4.9.0, httpx-0.30.0, cov-5.0.0, xdist-3.6.1, metadata-3.1.1 [e2e-llm-inference-service] asyncio: mode=Mode.STRICT [e2e-llm-inference-service] created: 2/2 workers [e2e-llm-inference-service] 2 workers [42 items] [e2e-llm-inference-service] [e2e-llm-inference-service] scheduling tests via WorkStealingScheduling [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] 2026-07-13 19:04:55.280 6134 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] 2026-07-13 19:04:55.280 6131 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] 2026-07-13 19:04:55.335 6131 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-13 19:04:55.335 6131 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-13 19:04:55.358 6131 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-13 19:04:55.358 6131 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-13 19:04:55.368 6131 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-13 19:04:55.368 6131 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] 2026-07-13 19:05:04.590 6131 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-13 19:05:04.590 6131 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-13 19:05:04.624 6131 kserve.trace ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-13 19:05:04.624 6131 kserve.trace INFO [gw_api.py:create_or_update_gateway():57] ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] 2026-07-13 19:14:07.781 6131 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-13 19:14:07.781 6131 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-13 19:14:07.827 6131 kserve.trace ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-13 19:14:07.827 6131 kserve.trace INFO [gw_api.py:create_or_update_gateway():57] ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] 2026-07-13 19:22:06.117 6131 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-13 19:22:06.117 6131 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-13 19:22:06.161 6131 kserve.trace ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-13 19:22:06.161 6131 kserve.trace INFO [gw_api.py:create_or_update_gateway():57] ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-13 19:22:06.161 6131 kserve.trace Checking HttpRoute router-route-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-13 19:22:06.161 6131 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-13 19:22:06.164 6131 kserve.trace Resource not found, creating HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-13 19:22:06.164 6131 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-13 19:22:06.175 6131 kserve.trace ✓ Successfully created HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-13 19:22:06.175 6131 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-13 19:22:06.176 6131 kserve.trace Checking HttpRoute router-route-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-13 19:22:06.176 6131 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-13 19:22:06.182 6131 kserve.trace Resource not found, creating HttpRoute router-route-2 [e2e-llm-inference-service] 2026-07-13 19:22:06.182 6131 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-2 [e2e-llm-inference-service] 2026-07-13 19:22:06.194 6131 kserve.trace ✓ Successfully created HttpRoute router-route-2 [e2e-llm-inference-service] 2026-07-13 19:22:06.194 6131 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-2 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] 2026-07-13 19:33:17.846 6131 kserve.trace Checking Gateway router-gateway-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-13 19:33:17.846 6131 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-13 19:33:17.876 6131 kserve.trace Resource not found, creating Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-13 19:33:17.876 6131 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-13 19:33:17.883 6131 kserve.trace ✓ Successfully created Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-13 19:33:17.883 6131 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-13 19:33:17.883 6131 kserve.trace Checking HttpRoute router-route-3 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-13 19:33:17.883 6131 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-3 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-13 19:33:17.887 6131 kserve.trace Resource not found, creating HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-13 19:33:17.887 6131 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-13 19:33:17.897 6131 kserve.trace ✓ Successfully created HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-13 19:33:17.897 6131 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-13 19:33:17.898 6131 kserve.trace Checking HttpRoute router-route-4 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-13 19:33:17.898 6131 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-4 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-13 19:33:17.903 6131 kserve.trace Resource not found, creating HttpRoute router-route-4 [e2e-llm-inference-service] 2026-07-13 19:33:17.903 6131 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-4 [e2e-llm-inference-service] 2026-07-13 19:33:17.911 6131 kserve.trace ✓ Successfully created HttpRoute router-route-4 [e2e-llm-inference-service] 2026-07-13 19:33:17.911 6131 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-4 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [e2e-llm-inference-service] =================================== FAILURES =================================== [e2e-llm-inference-service] ___ test_llm_inference_service[router-managed-workload-single-cpu-model-pvc] ___ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc... {'name': 'model-pvc-llmisvc-model-pvc-rou-7b41e44b'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[0], ROUTER_ROUTES[1]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[1], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[1]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[2], ROUTER_ROUTES[3]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[create_scheduler_configmap], [e2e-llm-inference-service] after_test=[delete_scheduler_configmap], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Precise prefix KV cache routing test [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="precise-prefix-cache-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name=f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] [e2e-llm-inference-service] test_case.llm_service.metadata.annotations[ [e2e-llm-inference-service] "security.opendatahub.io/enable-auth" [e2e-llm-inference-service] ] = "false" [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:812: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...c-mod-0fdbe7a1'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-7b41e44b'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-13T19:17:50.256298', start_time = 1783970270.2567408 [e2e-llm-inference-service] duration = 900.5042035579681, timestamp_end = '2026-07-13T19:32:50.760960' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....llmisvc-mod-0fdbe7a1'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-7b41e44b'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1204: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f604d03c860> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1215: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-13T19:18:06Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:17:53Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-13T19:18:25Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-13T19:18:25Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1199: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1749 Created PVC e2e-pvc-model-storage in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1887 Created model download Job e2e-pvc-model-download in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1911 Model download Job e2e-pvc-model-download completed successfully [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig router-managed-llmisvc-model-pv-6a7659e6 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-model-pv-6a7659e6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-model-pv-6a7659e6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig workload-single-cpu-llmisvc-mod-0fdbe7a1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig workload-single-cpu-llmisvc-mod-0fdbe7a1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig workload-single-cpu-llmisvc-mod-0fdbe7a1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-7b41e44b in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-7b41e44b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-7b41e44b [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-07-13T19:17:50.195521] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-model-pvc-router-manage-59b9d263', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-59b9d263', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-6a7659e6'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-0fdbe7a1'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-7b41e44b'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-13T19:17:50.208355] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-59b9d263', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-6a7659e6'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-0fdbe7a1'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-7b41e44b'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-13T19:17:50.256068] end - ✅ in 0.047s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-13T19:17:50.256298] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-59b9d263', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-6a7659e6'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-0fdbe7a1'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-7b41e44b'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-13T19:17:53Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-13T19:17:53Z', 'message': 'Inference Pool kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-13T19:17:53Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:17:53Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-13T19:17:53Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-13T19:17:53Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-13T19:17:53Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:17:53Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-13T19:18:06Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:17:53Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-13T19:18:06Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:17:53Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-13T19:18:25Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-13T19:18:25Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1219 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-13T19:18:06Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:17:53Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-13T19:18:25Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-13T19:18:25Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-13T19:32:50.760960] end - ❌ 900.504s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-13T19:18:06Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:17:53Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-13T19:18:25Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-13T19:18:25Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:831 [router-managed-workload-single-cpu-model-pvc] ❌ ERROR: Failed to call llm inference service llmisvc-model-pvc-router-manage-59b9d263: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-13T19:18:06Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:17:53Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-13T19:18:25Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-13T19:18:25Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1240 🔍 # Diagnostics for 'llmisvc-model-pvc-router-manage-59b9d263' in 'kserve-ci-e2e-test' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1241 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1242 # LLMInferenceService llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1245 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:17:50Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:17:50Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:17:50Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-13T19:18:25Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] resourceVersion: '36454' [e2e-llm-inference-service] uid: 7e02ecfc-f502-4ba5-b608-68b7d6fd73bb [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-llmisvc-model-pv-6a7659e6 [e2e-llm-inference-service] - name: workload-single-cpu-llmisvc-mod-0fdbe7a1 [e2e-llm-inference-service] - name: model-pvc-llmisvc-model-pvc-rou-7b41e44b [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://a7407f728462246e2be4fe1c58f85714-238314375.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a7407f728462246e2be4fe1c58f85714-238314375.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:18:06Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:18:06Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:18:06Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:17:53Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:18:06Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:18:25Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:18:25Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:18:06Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://a7407f728462246e2be4fe1c58f85714-238314375.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:44 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:45 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9 to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:45 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:45 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:45 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:31 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.45:8000/health": dial tcp 10.134.0.45:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:47 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:02 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.45:8000/health": dial tcp 10.134.0.45:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-ccd54d6d4-cgwk9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:42 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:42 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:42 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:43 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:43 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:43 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:47 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:47 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-router-scheduler-58f5ddbc8f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-ccd54d6d4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:36 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy auth-disabled-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "auth-disabled-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-disabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-disabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-disabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-disabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:42 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-disabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:08 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:08 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:08 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:08 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:11 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulling Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:38 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 27.071s (27.071s including waiting). Image size: 3531177328 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:38 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:38 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:18 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.41:8000/health": dial tcp 10.134.0.41:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:49 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.41:8000/health": context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-7dfddf4646-c84nz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:08 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:08 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:08 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:08 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:09 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:11 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.571s (1.571s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:11 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:11 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-7dfddf4646 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-enabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-enabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-enabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-enabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-enabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:28 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-enabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:07 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:07 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:07 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:53 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.43:8000/health": dial tcp 10.134.0.43:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:05 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:20 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.43:8000/health": dial tcp 10.134.0.43:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-58b5cf756b-27d2s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:04 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:04 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:04 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:04 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:05 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:05 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:05 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:05 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:05 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-router-scheduler-6c486c5958 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-58b5cf756b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-invalid-token-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-invalid-token-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-invalid-token-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-invalid-token-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:08 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-invalid-token-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:04 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-invalid-token-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.55/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:36 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:36 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:36 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:32:01 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.55:8001/health": dial tcp 10.134.0.55:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-6d9d59446f-spptn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.56/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:35 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:35 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:35 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:32:01 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.56:8000/health": dial tcp 10.134.0.56:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:33 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:33 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:33 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcc46c8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-6d9d59446f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:20 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-pd-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-pd-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:32:36 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-pd-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.51/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:36 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:36 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:36 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.51:8000/health": dial tcp 10.134.0.51:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:21:45 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-965ccd984-nft7b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6 to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:32 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:32 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:32 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:33 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:33 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:33 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:21:45 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:21:45 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-router-scheduler-685c77879f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-965ccd984 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:30 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:20:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:20:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:20:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:21:36 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:21:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/e2e-pvc-model-download-w2mvs to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:35 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-3823dadc-7c52-412c-8ef9-84caab00a904" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:39 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:39 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:39 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:39 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:29 kserve-ci-e2e-test job-controller Normal SuccessfulCreate Created pod: e2e-pvc-model-download-w2mvs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:48 kserve-ci-e2e-test job-controller Normal Completed Job completed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:29 kserve-ci-e2e-test persistentvolume-controller Normal WaitForFirstConsumer waiting for first consumer to be created before binding [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:29 kserve-ci-e2e-test persistentvolume-controller Normal ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:29 kserve-ci-e2e-test ebs.csi.aws.com_aws-ebs-csi-driver-controller-d8db6f98-dhbrp_42e66c8b-a583-47f3-9a2d-f39fd5be165e Normal Provisioning External provisioner is provisioning volume for claim "kserve-ci-e2e-test/e2e-pvc-model-storage" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:32 kserve-ci-e2e-test ebs.csi.aws.com_aws-ebs-csi-driver-controller-d8db6f98-dhbrp_42e66c8b-a583-47f3-9a2d-f39fd5be165e Normal ProvisioningSucceeded Successfully provisioned volume pvc-3823dadc-7c52-412c-8ef9-84caab00a904 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:02 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6 to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:03 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:03 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:04 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.206s (1.206s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:04 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:04 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:16 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:02 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:20 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:20 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:21 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:21 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:21 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:50 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:05 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.40:8000/health": EOF [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:20 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6 to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.53/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:03 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" in 778ms (778ms including waiting). Image size: 44914394 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:03 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:03 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:04 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:04 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:04 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:08 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:08 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:08 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:28:12 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.53:8001/health": dial tcp 10.134.0.53:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:28:49 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:28:49 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9 to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.54/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:06 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:06 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:06 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:28:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.54:8000/health": dial tcp 10.134.0.54:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:28:49 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f89477b49 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:01 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:28:43 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-50bc673d] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:28:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27 to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:00 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:00 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:00 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:18:46 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.49:8000/health": dial tcp 10.134.0.49:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:07 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:22 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.49:8000/health": dial tcp 10.134.0.49:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:54 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:18:57 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-87882a8e] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:15 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:16 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:16 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:16 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:31 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:15 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:13 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:26 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-e95b1dc1] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:24 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:24 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:24 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:39 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.47:8000/health": dial tcp 10.134.0.47:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:17 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:18 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler-6698955b74 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97 to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:20 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:20 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:20 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:20 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:20 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:17 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:17 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:18 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:18 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:33 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:33 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:50 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-7ca60146] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:52 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:53 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:53 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:58 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.48:8000/health": dial tcp 10.134.0.48:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:10 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-844f8ff647 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:49 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:49 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:49 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:49 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:50 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:50 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:50 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:09 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:09 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:47 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:08 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-ba4d693a] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:57 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-3823dadc-7c52-412c-8ef9-84caab00a904" Volume is already used by pod(s) llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-59b9d263-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-59b9d263-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testc9569cf4801efc0ed27b2f25ffaee875-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:18:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:27:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:18:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:32 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:48 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.38:8000/health": dial tcp 10.133.0.38:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:46 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:31 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-4f8c0978] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:42 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:42 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:42 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:15 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:39 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:14 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-a50492e9] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:52 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:07 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.38:8000/health": dial tcp 10.132.0.38:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:06 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-4b931143-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-4b931143-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:40 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-4b931143] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:01 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:35 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:01 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:58 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-5b1e8f15-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:34 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-5b1e8f15] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:33 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:48 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.35:8000/health": dial tcp 10.132.0.35:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:57 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-e45d1f79-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-e45d1f79-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:32 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-e45d1f79] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:56 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:56 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:33 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:33 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler-6fcb489785 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheduler-69d8c68fd8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:03 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:03 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:03 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:03 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:04 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:04 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:04 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:28:49 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:28:49 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-5dd88bfbb7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:00 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.205s (1.205s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:00 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:00 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:33 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:33 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:52 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:52 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler-5f555d4d85 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:52 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:52 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:52 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:52 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:57 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-3823dadc-7c52-412c-8ef9-84caab00a904" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler-688ff6d544 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5 to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:57 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:57 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:57 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:57 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:57 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:57 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:57 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:07 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:07 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:42 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:42 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:42 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:15 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:15 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler-6d86bd4d9d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-68cc9685d6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:08 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:08 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:08 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:08 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:52 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:52 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-749449dbc8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:35 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:35 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2 to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:18 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:19 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.48s (1.48s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:19 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:19 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:26 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" in 5.156s (5.156s including waiting). Image size: 300570036 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:23 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:25 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.796s (1.796s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:25 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:25 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:26 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:13 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-6ffc955574-v494f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:13 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-6ffc955574-p26g2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" in 4.75s (4.75s including waiting). Image size: 300570036 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:20 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:22 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.694s (1.694s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:52 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:52 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:22 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:52 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" in 30.148s (30.148s including waiting). Image size: 2989890188 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:52 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:52 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:52 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:05 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning Unhealthy Liveness probe failed: timeout: failed to connect service "10.132.0.34:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:26 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:26 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:26 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-d84595fb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:13 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-6ffc955574 from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:07 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:25 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [precise-prefix-cache-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:23 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:23 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:25 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" in 2.49s (2.49s including waiting). Image size: 181251556 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:26 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:26 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:32 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.133.0.33:15021/healthz/ready": dial tcp 10.133.0.33:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:26 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:08 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:56 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:56 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:59 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" in 3.513s (3.513s including waiting). Image size: 181251556 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:59 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:59 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:06 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:09 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:20 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:20 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:20 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.46:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.46:15021/healthz/ready": read tcp 10.132.0.2:35418->10.132.0.46:15021: read: connection reset by peer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.46:15021/healthz/ready": dial tcp 10.132.0.46:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:55 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-54c47cdfc7-lfprg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:06 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-54c47cdfc7-lfprg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:22 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-54c47cdfc7-kgtbh [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:08 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-54c47cdfc7-kgtbh [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:09 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-54c47cdfc7-r2bjb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69 to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:05 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:18 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:18 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:18 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:28 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:55 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:55 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:59 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" in 3.706s (3.706s including waiting). Image size: 181251556 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:59 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:59 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:03 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4 to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:06 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:25 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.45:15021/healthz/ready": dial tcp 10.132.0.45:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:28 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7 to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:08 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:08 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:08 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:08 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:10 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:55 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-kkqwb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:01 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-799f46c59b-kkqwb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:04 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-fqc69 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:28 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-799f46c59b-fqc69 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-s6nj7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:08 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-799f46c59b-s6nj7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:06 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-rbgv4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:27 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-799f46c59b-rbgv4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:06 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-799f46c59b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:55 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:08 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-54c47cdfc7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:59 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:27 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set router-gateway-1-openshift-default-799f46c59b from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:08 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set router-gateway-1-openshift-default-54c47cdfc7 from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:06 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.52/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:30 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:30 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:30 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:24:16 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.52:8000/health": dial tcp 10.134.0.52:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:25:31 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:25:47 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.52:8000/health": dial tcp 10.134.0.52:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:25 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-555479964d-4kccs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:28 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:28 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:28 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:25:31 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:25:31 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-router-scheduler-7f865578d7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:25 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-555479964d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:24:29 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [router-with-refs-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:25:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:204 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': 'b0f20e5e-6ea4-4b9b-b7b7-dd3c1dabe0bd', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Mon, 13 Jul 2026 19:32:51 GMT', 'Content-Length': '247'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f\" is waiting to start: ContainerCreating","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 {"level":"info","ts":1783970272.5845404,"logger":"setup","caller":"runner/runner.go:196","msg":"GIE build","commit-sha":"181aa8358916e19b8844ccc752b2d6153d4b2ad6","build-ref":"v0.9.0-rc.2"} [e2e-llm-inference-service] Flag --model-server-metrics-scheme has been deprecated, This flag is deprecated. Configure via EndpointPickerConfig data layer plugin parameters instead. [e2e-llm-inference-service] {"level":"info","ts":1783970272.5846655,"logger":"setup","caller":"runner/runner.go:217","msg":"Flags processed","flags":{"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"https","pool-group":"inference.networking.k8s.io","pool-name":"llmisvc-model-pvc-router-manage-59b9d263-inference-pool","pool-namespace":"kserve-ci-e2e-test","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783970272.5847874,"logger":"setup.trace","caller":"tracing/telemetry.go:123","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.5851817,"caller":"loader/configloader.go:89","msg":"DEPRECATION: apiVersion inference.networking.x-k8s.io/v1alpha1/EndpointPickerConfig is deprecated","replacement":"llm-d.ai/v1alpha1/EndpointPickerConfig"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.5852137,"caller":"loader/configloader.go:121","msg":"Loaded raw configuration","config":"{Plugins: [{Type: single-profile-handler} {Type: queue-scorer} {Type: prefix-cache-scorer} {Type: max-score-picker}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}]}"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.5852242,"logger":"setup","caller":"runner/runner.go:622","msg":"Data layer: ENABLED"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.5855427,"logger":"setup","caller":"runner/runner.go:281","msg":"Raw config after phase one","config":{"apiVersion":"inference.networking.x-k8s.io/v1alpha1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"single-profile-handler","parameters":null,"type":"single-profile-handler"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"}],"schedulingProfiles":[{"name":"default","plugins":[{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]}]}} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6210816,"logger":"utilization-detector/utilization-detector","caller":"utilization/detector.go:83","msg":"Creating new UtilizationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms","headroom":0} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6212015,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"vllm","mapping":"Mapping{all specs enabled}"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6212451,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"sglang","mapping":"Mapping{disabled: [lora]}"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6212842,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"trtllm-serve","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6214006,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton-tensorrt-llm","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6214228,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton","mapping":"Mapping{disabled: [kv, lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6214843,"caller":"loader/configloader.go:154","msg":"Instantiated all plugins and applied system defaults. Effective raw configuration","config":"{Plugins: [{Name: single-profile-handler, Type: single-profile-handler} {Name: queue-scorer, Type: queue-scorer} {Name: prefix-cache-scorer, Type: prefix-cache-scorer} {Name: max-score-picker, Type: max-score-picker} {Name: fcfs-ordering-policy, Type: fcfs-ordering-policy} {Name: global-strict-fairness-policy, Type: global-strict-fairness-policy} {Name: static-usage-limit-policy, Type: static-usage-limit-policy} {Name: openai-parser, Type: openai-parser} {Name: anthropic-parser, Type: anthropic-parser} {Name: vllmhttp-parser, Type: vllmhttp-parser} {Name: utilization-detector, Type: utilization-detector} {Name: metrics-data-source, Type: metrics-data-source} {Name: core-metrics-extractor, Type: core-metrics-extractor}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}], DataLayer: {Sources: [{PluginRef: metrics-data-source, Extractors: [{PluginRef: core-metrics-extractor}]}], Discovery: }, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}]}}"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6215403,"caller":"approximateprefix/plugin.go:88","msg":"Prefix DataProducer initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":2048,"maxPrefixTokensToMatch":131072,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6216168,"caller":"approximateprefix/plugin.go:111","msg":"WARNING: configured blockSizeTokens is below the recommended minimum, overriding it.","blockSizeTokens":16,"minimum":64,"issue":"https://github.com/llm-d/llm-d-router/issues/1158"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6216419,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"approx-prefix-cache-producer/approx-prefix-cache-producer","dataKey":"PrefixCacheMatchInfoDataKey/approx-prefix-cache-producer","consumer":"prefix-cache-scorer"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6216955,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"token-producer/token-producer","dataKey":"TokenizedPrompt/token-producer","consumer":"approx-prefix-cache-producer"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6217961,"caller":"runner/runner.go:685","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6218119,"logger":"setup","caller":"runner/runner.go:308","msg":"EPP config after phase two","config":"{SchedulerConfig:{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker}]} SaturationDetector:0xc000764680 DataConfig:{Sources:[{Plugin:0xc0006a6f30 Extractors:[0xc000764980]}]} FlowControlConfig: ParserRegistry:0xc000764f80}"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6744912,"logger":"setup","caller":"runner/runner.go:352","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6745393,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/symbol"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.674561,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.674568,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6745734,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6745796,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6745856,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6745913,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6745973,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/cmdline"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6746047,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/profile"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.674611,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/trace"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.674619,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6746314,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/plugins/state"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.674644,"logger":"setup","caller":"runner/runner.go:373","msg":"parsed config","scheduler-config":"{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.674682,"logger":"setup","caller":"datalayer/runtime.go:99","msg":"Configuring datalayer runtime","numSources":1} [e2e-llm-inference-service] {"level":"info","ts":1783970272.674696,"logger":"setup","caller":"datalayer/runtime.go:118","msg":"Processing source","source":"metrics-data-source","numExtractors":1} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6747205,"logger":"setup","caller":"datalayer/runtime.go:147","msg":"Source configured","source":"metrics-data-source","extractors":["core-metrics-extractor/core-metrics-extractor"]} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6747403,"logger":"setup","caller":"datalayer/runtime.go:206","msg":"Datalayer runtime configured","pollers":1,"notifiers":0,"endpointSources":0} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6747541,"logger":"setup","caller":"runner/runner.go:833","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6748536,"logger":"setup","caller":"runner/runner.go:721","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6748714,"logger":"setup","caller":"runner/runner.go:260","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6749046,"logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6751742,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6752746,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6753325,"logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":1783970272.675548,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.675742,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6758947,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6762333,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.6762784,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.676432,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":1783970272.7158234,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.7169206,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.7172365,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.7174685,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.7766314,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.7766726,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783970272.7766316,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.77676,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783970272.8769968,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.876992,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":1783970272.8770678,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783970272.8771226,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783970272.8774087,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-59b9d263-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-pvc-router-manage-59b9d263-inference-pool","reconcileID":"56eae22d-3736-4c84-8d34-346ed0b506ff"} [e2e-llm-inference-service] {"level":"info","ts":1783970285.5652366,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-59b9d263-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-pvc-router-manage-59b9d263-inference-pool","reconcileID":"12c4efe1-e4a7-4c01-abba-af65d43191cc"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 6c33418f-2e2f-44d4-a285-e790ba24f6f7 [e2e-llm-inference-service] resourceVersion: '36430' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-13T19:18:23Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:18:23Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.133.0.41 [e2e-llm-inference-service] nodeName: ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff [e2e-llm-inference-service] uid: 42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 6a08219c-fd76-4f5a-a414-4abeb00bc600 [e2e-llm-inference-service] resourceVersion: '35915' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 634b78c4-751c-4880-9f6c-1ea69b737987 [e2e-llm-inference-service] resourceVersion: '35928' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 585587bc9d [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.50/23"],"mac_address":"0a:58:0a:86:00:32","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.50/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d [e2e-llm-inference-service] uid: a317bc10-2176-4c0d-8ebd-65bc9a349adf [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-128-215 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a317bc10-2176-4c0d-8ebd-65bc9a349adf"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-4ghrx [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"facebook/opt-125m\"\ [e2e-llm-inference-service] \ \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\" \\\n --port 8000\ [e2e-llm-inference-service] \ \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n --enable-ssl-refresh\ [e2e-llm-inference-service] \ \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-4ghrx [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-gv2sm [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] hostIP: 10.0.128.215 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.128.215 [e2e-llm-inference-service] startTime: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: ContainerCreating [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-4ghrx [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff [e2e-llm-inference-service] generateName: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler-5f555d4d85- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446 [e2e-llm-inference-service] resourceVersion: '36429' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 5f555d4d85 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.41/23"],"mac_address":"0a:58:0a:85:00:29","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.41/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.41\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:29\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler-5f555d4d85 [e2e-llm-inference-service] uid: e1ed0f0a-acac-4ece-ae95-668b801421f6 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-139-134 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e1ed0f0a-acac-4ece-ae95-668b801421f6"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:18:23Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.41"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-zgccc [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: prefix-cache-scorer\n\ [e2e-llm-inference-service] - type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-zgccc [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-gv2sm [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-d8pk7 [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:18:23Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:18:23Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] hostIP: 10.0.139.134 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.139.134 [e2e-llm-inference-service] podIP: 10.133.0.41 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.41 [e2e-llm-inference-service] startTime: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-router-endpoint-picker@sha256:06b6c75d77afd0e07053402752a9736c2dfbc12a306d0d37d963aac4c1d4e6a6 [e2e-llm-inference-service] containerID: cri-o://315e85ec70223bfce2eeaa66dcdb3e7c586c1cde9778a18356dc1ab7e750c808 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-zgccc [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 2899c6d4-9d31-4e02-8e61-4bd5f90753ad [e2e-llm-inference-service] resourceVersion: '35932' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-d8pk7 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: 7e02ecfc-f502-4ba5-b608-68b7d6fd73bb [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-d8pk7"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7e02ecfc-f502-4ba5-b608-68b7d6fd73bb"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-gv2sm"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-gv2sm [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-d8pk7 [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-gv2sm [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-d8pk7 [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 41cf1f17-e958-44d8-8b0d-e6b4ce1c72c5 [e2e-llm-inference-service] resourceVersion: '35945' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: 7e02ecfc-f502-4ba5-b608-68b7d6fd73bb [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7e02ecfc-f502-4ba5-b608-68b7d6fd73bb"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.222.231 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.222.231 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 69db744f-8222-4c63-8cae-1387c6094348 [e2e-llm-inference-service] resourceVersion: '35913' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: 7e02ecfc-f502-4ba5-b608-68b7d6fd73bb [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7e02ecfc-f502-4ba5-b608-68b7d6fd73bb"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.17.209 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.17.209 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: c01ec407-ed73-4ffc-a125-52d36e9dde52 [e2e-llm-inference-service] resourceVersion: '43683' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: 7e02ecfc-f502-4ba5-b608-68b7d6fd73bb [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7e02ecfc-f502-4ba5-b608-68b7d6fd73bb"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-13T19:27:52Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:unavailableReplicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] unavailableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-13T19:27:52Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:27:52Z' [e2e-llm-inference-service] reason: ProgressDeadlineExceeded [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d" [e2e-llm-inference-service] has timed out progressing. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: f03787a2-4122-494a-9b4b-43a0090ac5ed [e2e-llm-inference-service] resourceVersion: '36433' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: 7e02ecfc-f502-4ba5-b608-68b7d6fd73bb [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7e02ecfc-f502-4ba5-b608-68b7d6fd73bb"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-13T19:18:23Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n-\ [e2e-llm-inference-service] \ name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-13T19:18:23Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:18:23Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-13T19:18:23Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler-5f555d4d85" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: a317bc10-2176-4c0d-8ebd-65bc9a349adf [e2e-llm-inference-service] resourceVersion: '35924' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 585587bc9d [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve [e2e-llm-inference-service] uid: c01ec407-ed73-4ffc-a125-52d36e9dde52 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c01ec407-ed73-4ffc-a125-52d36e9dde52"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 585587bc9d [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 585587bc9d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler-5f555d4d85 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: e1ed0f0a-acac-4ece-ae95-668b801421f6 [e2e-llm-inference-service] resourceVersion: '36432' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 5f555d4d85 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler [e2e-llm-inference-service] uid: f03787a2-4122-494a-9b4b-43a0090ac5ed [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"f03787a2-4122-494a-9b4b-43a0090ac5ed"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-13T19:18:23Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 5f555d4d85 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 5f555d4d85 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n-\ [e2e-llm-inference-service] \ name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: ad7d8895-712e-46eb-b57f-e2999759b732 [e2e-llm-inference-service] resourceVersion: '35939' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: 7e02ecfc-f502-4ba5-b608-68b7d6fd73bb [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7e02ecfc-f502-4ba5-b608-68b7d6fd73bb"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 7ce2c66e-911f-4e42-9790-11d019cffb5f [e2e-llm-inference-service] resourceVersion: '35937' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: 7e02ecfc-f502-4ba5-b608-68b7d6fd73bb [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7e02ecfc-f502-4ba5-b608-68b7d6fd73bb"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-service-45zqg [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-59b9d263-epp-service- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: d854e582-3112-4460-a704-b01925c2ece0 [e2e-llm-inference-service] resourceVersion: '36431' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-59b9d263-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-13T19:18:23Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-service [e2e-llm-inference-service] uid: 41cf1f17-e958-44d8-8b0d-e6b4ce1c72c5 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-13T19:18:23Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"41cf1f17-e958-44d8-8b0d-e6b4ce1c72c5"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.133.0.41 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff [e2e-llm-inference-service] uid: 42f8b4d4-e92d-4c0a-8f47-06c3ec4ad446 [e2e-llm-inference-service] nodeName: ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-sn8l56 [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 4aa14de2-ec4a-4b33-b382-03af4bda4556 [e2e-llm-inference-service] resourceVersion: '35916' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] uid: 69db744f-8222-4c63-8cae-1387c6094348 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"69db744f-8222-4c63-8cae-1387c6094348"}: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: null [e2e-llm-inference-service] ports: null [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: ad7d8895-712e-46eb-b57f-e2999759b732 [e2e-llm-inference-service] resourceVersion: '35939' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: 7e02ecfc-f502-4ba5-b608-68b7d6fd73bb [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7e02ecfc-f502-4ba5-b608-68b7d6fd73bb"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 7ce2c66e-911f-4e42-9790-11d019cffb5f [e2e-llm-inference-service] resourceVersion: '35937' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: 7e02ecfc-f502-4ba5-b608-68b7d6fd73bb [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-13T19:17:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7e02ecfc-f502-4ba5-b608-68b7d6fd73bb"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7e02ecfc-f502-4ba5-b608-68b7d6fd73bb"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:18:05Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-13T19:18:05Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-13T19:18:06Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: 7e02ecfc-f502-4ba5-b608-68b7d6fd73bb [e2e-llm-inference-service] resourceVersion: '36235' [e2e-llm-inference-service] uid: 71f01b0c-dcd3-4315-86ce-dba37d166787 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:18:05Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7e02ecfc-f502-4ba5-b608-68b7d6fd73bb"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:18:05Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-13T19:18:05Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-13T19:18:06Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: 7e02ecfc-f502-4ba5-b608-68b7d6fd73bb [e2e-llm-inference-service] resourceVersion: '36235' [e2e-llm-inference-service] uid: 71f01b0c-dcd3-4315-86ce-dba37d166787 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:18:05Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7e02ecfc-f502-4ba5-b608-68b7d6fd73bb"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-13T19:18:05Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: 7e02ecfc-f502-4ba5-b608-68b7d6fd73bb [e2e-llm-inference-service] resourceVersion: '36220' [e2e-llm-inference-service] uid: 7355118f-ca0b-4e26-95e7-c2abe833ff83 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:18:05Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:18:05Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7e02ecfc-f502-4ba5-b608-68b7d6fd73bb"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-13T19:17:54Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-route-authn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: 7e02ecfc-f502-4ba5-b608-68b7d6fd73bb [e2e-llm-inference-service] resourceVersion: '36067' [e2e-llm-inference-service] uid: d9e2eff5-88ef-487d-a832-254c17d5679a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:17:53Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:17:54Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7e02ecfc-f502-4ba5-b608-68b7d6fd73bb"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: 7e02ecfc-f502-4ba5-b608-68b7d6fd73bb [e2e-llm-inference-service] resourceVersion: '36011' [e2e-llm-inference-service] uid: 3ea2ab3e-113e-45bc-888e-1702c3d27fd1 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-59b9d263-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-59b9d263-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:18:05Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7e02ecfc-f502-4ba5-b608-68b7d6fd73bb"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:18:05Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: 7e02ecfc-f502-4ba5-b608-68b7d6fd73bb [e2e-llm-inference-service] resourceVersion: '36224' [e2e-llm-inference-service] uid: b0c61144-706a-4147-89d2-7465799eac36 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-59b9d263-inference--ip-737e880e.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:17:53Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7e02ecfc-f502-4ba5-b608-68b7d6fd73bb"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:17:53Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: 7e02ecfc-f502-4ba5-b608-68b7d6fd73bb [e2e-llm-inference-service] resourceVersion: '36015' [e2e-llm-inference-service] uid: b8ac32c6-1f17-4829-a555-f4494eb30a1d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7e02ecfc-f502-4ba5-b608-68b7d6fd73bb"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: 7e02ecfc-f502-4ba5-b608-68b7d6fd73bb [e2e-llm-inference-service] resourceVersion: '36011' [e2e-llm-inference-service] uid: 3ea2ab3e-113e-45bc-888e-1702c3d27fd1 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-59b9d263-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-59b9d263-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:18:05Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7e02ecfc-f502-4ba5-b608-68b7d6fd73bb"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:18:05Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: 7e02ecfc-f502-4ba5-b608-68b7d6fd73bb [e2e-llm-inference-service] resourceVersion: '36224' [e2e-llm-inference-service] uid: b0c61144-706a-4147-89d2-7465799eac36 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-59b9d263-inference--ip-737e880e.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:17:53Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7e02ecfc-f502-4ba5-b608-68b7d6fd73bb"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:17:53Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: 7e02ecfc-f502-4ba5-b608-68b7d6fd73bb [e2e-llm-inference-service] resourceVersion: '36015' [e2e-llm-inference-service] uid: b8ac32c6-1f17-4829-a555-f4494eb30a1d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7e02ecfc-f502-4ba5-b608-68b7d6fd73bb"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: 7e02ecfc-f502-4ba5-b608-68b7d6fd73bb [e2e-llm-inference-service] resourceVersion: '36011' [e2e-llm-inference-service] uid: 3ea2ab3e-113e-45bc-888e-1702c3d27fd1 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-59b9d263-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-59b9d263-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:18:05Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7e02ecfc-f502-4ba5-b608-68b7d6fd73bb"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:18:05Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: 7e02ecfc-f502-4ba5-b608-68b7d6fd73bb [e2e-llm-inference-service] resourceVersion: '36224' [e2e-llm-inference-service] uid: b0c61144-706a-4147-89d2-7465799eac36 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-59b9d263-inference--ip-737e880e.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:17:53Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7e02ecfc-f502-4ba5-b608-68b7d6fd73bb"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:17:53Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: 7e02ecfc-f502-4ba5-b608-68b7d6fd73bb [e2e-llm-inference-service] resourceVersion: '36015' [e2e-llm-inference-service] uid: b8ac32c6-1f17-4829-a555-f4494eb30a1d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7e02ecfc-f502-4ba5-b608-68b7d6fd73bb"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:17:52Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: 7e02ecfc-f502-4ba5-b608-68b7d6fd73bb [e2e-llm-inference-service] resourceVersion: '35954' [e2e-llm-inference-service] uid: 701bea62-58b9-40cf-9975-f2958233f156 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:32:52Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 5f555d4d85 [e2e-llm-inference-service] timestamp: '2026-07-13T19:32:47Z' [e2e-llm-inference-service] window: 16.827s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 744874n [e2e-llm-inference-service] memory: 24180Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-07-13T19:32:52.654370] end - ❌ 902.458s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-13T19:18:06Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:17:53Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-13T19:18:25Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-13T19:18:25Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:18:06Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] _____ test_llm_inference_service[router-managed-workload-pd-cpu-model-pvc] _____ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-pd-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-mod... {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[0], ROUTER_ROUTES[1]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[1], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[1]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[2], ROUTER_ROUTES[3]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[create_scheduler_configmap], [e2e-llm-inference-service] after_test=[delete_scheduler_configmap], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Precise prefix KV cache routing test [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="precise-prefix-cache-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name=f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] [e2e-llm-inference-service] test_case.llm_service.metadata.annotations[ [e2e-llm-inference-service] "security.opendatahub.io/enable-auth" [e2e-llm-inference-service] ] = "false" [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:812: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...del-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-13T19:32:53.738099', start_time = 1783971173.738436 [e2e-llm-inference-service] duration = 900.5924067497253, timestamp_end = '2026-07-13T19:47:54.330857' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....svc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1204: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f604d03cf40> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1215: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-13T19:34:01Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-13T19:34:01Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-13T19:34:01Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-13T19:34:01Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-13T19:34:01Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1199: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1752 PVC e2e-pvc-model-storage already exists in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1962 Model download Job already completed, skipping [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-07-13T19:32:53.460729] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-pd-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-model-pvc-router-manage-e8706282', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-13T19:32:53.473411] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-13T19:32:53.737963] end - ✅ in 0.264s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-13T19:32:53.738099] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-13T19:33:26Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'message': 'Inference Pool kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-13T19:34:01Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-13T19:34:01Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-13T19:34:01Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-13T19:34:01Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-13T19:34:01Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1219 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-13T19:34:01Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-13T19:34:01Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-13T19:34:01Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-13T19:34:01Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-13T19:34:01Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-13T19:47:54.330857] end - ❌ 900.592s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-13T19:34:01Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-13T19:34:01Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-13T19:34:01Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-13T19:34:01Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-13T19:34:01Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:831 [router-managed-workload-pd-cpu-model-pvc] ❌ ERROR: Failed to call llm inference service llmisvc-model-pvc-router-manage-e8706282: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-13T19:34:01Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-13T19:34:01Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-13T19:34:01Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-13T19:34:01Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-13T19:34:01Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1240 🔍 # Diagnostics for 'llmisvc-model-pvc-router-manage-e8706282' in 'kserve-ci-e2e-test' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1241 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1242 # LLMInferenceService llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1245 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:32:53Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:32:53Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:32:59Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:prefill: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-13T19:34:02Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] resourceVersion: '48514' [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] - name: workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] - name: model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://a7407f728462246e2be4fe1c58f85714-238314375.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a7407f728462246e2be4fe1c58f85714-238314375.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:34:01Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:34:01Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:33:26Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:33:26Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: PrefillWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:33:26Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:34:01Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:34:01Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:34:01Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:33:26Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://a7407f728462246e2be4fe1c58f85714-238314375.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:44 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:45 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9 to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:45 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:45 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:45 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:31 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.45:8000/health": dial tcp 10.134.0.45:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:47 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:02 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.45:8000/health": dial tcp 10.134.0.45:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-ccd54d6d4-cgwk9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:42 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:42 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:42 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:43 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:43 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:43 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:47 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:47 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-router-scheduler-58f5ddbc8f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-ccd54d6d4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:36 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy auth-disabled-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "auth-disabled-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-disabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-disabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-disabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-disabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:42 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-disabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:08 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:08 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:08 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:08 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:11 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulling Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:38 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 27.071s (27.071s including waiting). Image size: 3531177328 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:38 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:38 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:18 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.41:8000/health": dial tcp 10.134.0.41:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:49 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.41:8000/health": context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-7dfddf4646-c84nz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:08 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:08 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:08 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:08 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:09 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:11 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.571s (1.571s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:11 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:11 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-7dfddf4646 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-enabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-enabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-enabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-enabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-enabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:28 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-enabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:07 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:07 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:07 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:53 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.43:8000/health": dial tcp 10.134.0.43:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:05 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:20 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.43:8000/health": dial tcp 10.134.0.43:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-58b5cf756b-27d2s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:04 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:04 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:04 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:04 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:05 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:05 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:05 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:05 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:05 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-router-scheduler-6c486c5958 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-58b5cf756b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-invalid-token-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-invalid-token-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-invalid-token-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-invalid-token-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:08 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-invalid-token-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:04 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-invalid-token-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.55/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:36 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:36 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:36 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:32:01 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.55:8001/health": dial tcp 10.134.0.55:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:32:59 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:32:59 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-6d9d59446f-spptn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.56/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:35 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:35 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:35 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:32:01 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.56:8000/health": dial tcp 10.134.0.56:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:32:59 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:33 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:33 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:33 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:32:59 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:32:59 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcc46c8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-6d9d59446f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:20 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-pd-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-pd-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:32:36 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-pd-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:32:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.51/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:36 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:36 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:36 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.51:8000/health": dial tcp 10.134.0.51:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:21:45 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-965ccd984-nft7b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6 to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:32 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:32 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:32 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:33 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:33 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:33 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:21:45 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:21:45 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-router-scheduler-685c77879f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-965ccd984 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:30 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:20:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:20:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:20:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:21:36 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:21:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/e2e-pvc-model-download-w2mvs to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:35 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-3823dadc-7c52-412c-8ef9-84caab00a904" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:39 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:39 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:39 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:39 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:29 kserve-ci-e2e-test job-controller Normal SuccessfulCreate Created pod: e2e-pvc-model-download-w2mvs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:48 kserve-ci-e2e-test job-controller Normal Completed Job completed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:29 kserve-ci-e2e-test persistentvolume-controller Normal WaitForFirstConsumer waiting for first consumer to be created before binding [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:29 kserve-ci-e2e-test persistentvolume-controller Normal ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:29 kserve-ci-e2e-test ebs.csi.aws.com_aws-ebs-csi-driver-controller-d8db6f98-dhbrp_42e66c8b-a583-47f3-9a2d-f39fd5be165e Normal Provisioning External provisioner is provisioning volume for claim "kserve-ci-e2e-test/e2e-pvc-model-storage" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:32 kserve-ci-e2e-test ebs.csi.aws.com_aws-ebs-csi-driver-controller-d8db6f98-dhbrp_42e66c8b-a583-47f3-9a2d-f39fd5be165e Normal ProvisioningSucceeded Successfully provisioned volume pvc-3823dadc-7c52-412c-8ef9-84caab00a904 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:02 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6 to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:03 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:03 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:04 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.206s (1.206s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:04 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:04 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:16 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:02 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:20 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:20 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:21 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:21 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:21 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:50 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:05 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.40:8000/health": EOF [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:20 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:52 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:53 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.61/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:53 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:53 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:53 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:57 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:57 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:57 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:39:43 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.61:8000/health": dial tcp 10.134.0.61:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:40:46 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:02 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.61:8000/health": dial tcp 10.134.0.61:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:52 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:51 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-3c960099-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-3c960099-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:38:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:40:24 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-3c960099] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6 to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.53/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:03 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" in 778ms (778ms including waiting). Image size: 44914394 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:03 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:03 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:04 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:04 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:04 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:08 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:08 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:08 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:28:12 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.53:8001/health": dial tcp 10.134.0.53:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:28:49 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:28:49 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9 to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.54/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:06 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:06 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:06 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:28:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.54:8000/health": dial tcp 10.134.0.54:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:28:49 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f89477b49 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:01 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:28:43 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-50bc673d] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:28:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27 to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:00 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:00 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:00 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:18:46 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.49:8000/health": dial tcp 10.134.0.49:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:07 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:22 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.49:8000/health": dial tcp 10.134.0.49:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:54 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:18:57 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-87882a8e] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.55/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:31 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:31 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:31 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:35 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulling Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:42:02 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 27.76s (27.76s including waiting). Image size: 3531177328 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:42:03 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:42:03 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 in StatefulSet llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.62/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:31 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:31 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:34 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:34 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:34 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:43:00 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.62:8000/health": dial tcp 10.134.0.62:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test leaderworkerset Normal CreatingRevision Creating revision with key 848c75bc96 for a newly created LeaderWorkerSet [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created leader statefulset llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Replicas are progressing, with 0 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 in StatefulSet llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created worker statefulset for leader pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:43:10 kserve-ci-e2e-test leaderworkerset Normal AllGroupsReady All replicas are ready, with 1 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:15 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-scc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.LeaderWorkerSet kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test386bd5808c5c450f11fd8632e1f821fb-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:15 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:16 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:16 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:16 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:31 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:15 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:13 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:26 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-e95b1dc1] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:24 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:24 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:24 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:39 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.47:8000/health": dial tcp 10.134.0.47:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:17 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:18 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler-6698955b74 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97 to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:20 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:20 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:20 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:20 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:20 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:17 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:17 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:18 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:18 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:33 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:33 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:50 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-7ca60146] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:52 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:53 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:53 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:58 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.48:8000/health": dial tcp 10.134.0.48:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:10 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-844f8ff647 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:49 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:49 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:49 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:49 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:50 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:50 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:50 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:09 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:09 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:47 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:08 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-ba4d693a] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:57 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-3823dadc-7c52-412c-8ef9-84caab00a904" Volume is already used by pod(s) llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-59b9d263-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-59b9d263-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testc9569cf4801efc0ed27b2f25ffaee875-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:18:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:47:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:18:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:21 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:22 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:22 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79497db4cc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:21 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:03 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-e8706282-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testf5d060a5eb39a04e074b78907a1556a6-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:34:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:43:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:34:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:32 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:48 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.38:8000/health": dial tcp 10.133.0.38:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:46 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:31 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-4f8c0978] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:42 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:42 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:42 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:15 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:39 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:14 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-a50492e9] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:52 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:07 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.38:8000/health": dial tcp 10.132.0.38:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:06 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-4b931143-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-4b931143-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:40 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-4b931143] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:01 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:35 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:01 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:58 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-5b1e8f15-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:34 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-5b1e8f15] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:33 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:48 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.35:8000/health": dial tcp 10.132.0.35:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:57 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-e45d1f79-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-e45d1f79-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:32 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-e45d1f79] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:56 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:56 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:33 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:33 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler-6fcb489785 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheduler-6b4848c4bd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.54/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:31 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:31 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:32 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:32 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:32 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheduler-69d8c68fd8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:03 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:03 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:03 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:03 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:04 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:04 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:04 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:28:49 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:28:49 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-5dd88bfbb7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:00 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.205s (1.205s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:00 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:00 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:33 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:33 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:52 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:52 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler-5f555d4d85 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:52 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:52 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:52 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:52 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:57 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-3823dadc-7c52-412c-8ef9-84caab00a904" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler-688ff6d544 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5 to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:57 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:57 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:57 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:57 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:57 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:57 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:57 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:07 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:07 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:42 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:42 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:42 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:15 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:15 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler-6d86bd4d9d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:23 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:23 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-67b4bb9646 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:23 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-3823dadc-7c52-412c-8ef9-84caab00a904" Volume is already used by pod(s) llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:23 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.52/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:23 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:23 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:23 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-68cc9685d6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:08 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:08 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:08 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:08 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:52 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:52 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-749449dbc8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:35 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:35 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2 to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:18 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:19 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.48s (1.48s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:19 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:19 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:26 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" in 5.156s (5.156s including waiting). Image size: 300570036 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:23 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:25 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.796s (1.796s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:25 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:25 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:26 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:13 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-6ffc955574-v494f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:13 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-6ffc955574-p26g2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" in 4.75s (4.75s including waiting). Image size: 300570036 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:20 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:22 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.694s (1.694s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:52 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:52 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:22 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:52 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" in 30.148s (30.148s including waiting). Image size: 2989890188 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:52 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:52 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:52 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:05 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning Unhealthy Liveness probe failed: timeout: failed to connect service "10.132.0.34:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:26 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:26 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:26 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-d84595fb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:13 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-6ffc955574 from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:07 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:25 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [precise-prefix-cache-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:23 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:23 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:25 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" in 2.49s (2.49s including waiting). Image size: 181251556 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:26 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:26 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:32 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.133.0.33:15021/healthz/ready": dial tcp 10.133.0.33:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:26 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:08 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:56 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:56 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:59 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" in 3.513s (3.513s including waiting). Image size: 181251556 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:59 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:59 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:06 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:09 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:20 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:20 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:20 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.46:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.46:15021/healthz/ready": read tcp 10.132.0.2:35418->10.132.0.46:15021: read: connection reset by peer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.46:15021/healthz/ready": dial tcp 10.132.0.46:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:55 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-54c47cdfc7-lfprg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:06 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-54c47cdfc7-lfprg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:22 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-54c47cdfc7-kgtbh [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:08 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-54c47cdfc7-kgtbh [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:09 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-54c47cdfc7-r2bjb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69 to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:05 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:18 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:18 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:18 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:28 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:55 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:55 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:59 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" in 3.706s (3.706s including waiting). Image size: 181251556 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:59 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:59 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:03 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4 to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:06 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:25 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.45:15021/healthz/ready": dial tcp 10.132.0.45:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:28 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7 to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:08 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:08 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:08 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:08 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:10 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:55 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-kkqwb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:01 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-799f46c59b-kkqwb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:04 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-fqc69 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:28 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-799f46c59b-fqc69 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-s6nj7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:08 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-799f46c59b-s6nj7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:06 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-rbgv4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:27 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-799f46c59b-rbgv4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:06 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-799f46c59b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:55 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:08 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-54c47cdfc7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:59 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:27 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set router-gateway-1-openshift-default-799f46c59b from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:08 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set router-gateway-1-openshift-default-54c47cdfc7 from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:06 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28 to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:18 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.50/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:18 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:18 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:18 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:23 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:17 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-2-openshift-default-54c789bdc6-chk28 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:23 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-2-openshift-default-54c789bdc6-chk28 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:21 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.51/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:32 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:32 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:32 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:37 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.51:15021/healthz/ready": dial tcp 10.132.0.51:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:20 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-2-openshift-default-684dc9b4b9-c9txx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:17 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-2-openshift-default-54c789bdc6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:17 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:20 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-2-openshift-default-684dc9b4b9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:22 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:23 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set router-gateway-2-openshift-default-54c789bdc6 from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:17 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6 to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.59/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:51 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:51 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:51 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:55 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:55 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:55 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:36:30 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.59:8001/health": dial tcp 10.134.0.59:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:12 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:12 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-664bd66db5-cjjg6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.60/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:54 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:54 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:54 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:36:30 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.60:8000/health": dial tcp 10.134.0.60:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:12 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-prefill-9d68b57d4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:51 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.53/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:51 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:51 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:51 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:52 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:52 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:52 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:12 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:12 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-664bd66db5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-pd-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:34:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:36:50 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [router-with-refs-pd-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.52/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:30 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:30 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:30 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:24:16 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.52:8000/health": dial tcp 10.134.0.52:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:25:31 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:25:47 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.52:8000/health": dial tcp 10.134.0.52:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:25 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-555479964d-4kccs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:28 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:28 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:28 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:25:31 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:25:31 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-router-scheduler-7f865578d7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:25 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-555479964d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:24:29 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [router-with-refs-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:25:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### init-container 'llm-d-routing-sidecar' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:204 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '758eff00-3d35-4da6-8ff0-3701502aa4c1', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Mon, 13 Jul 2026 19:47:55 GMT', 'Content-Length': '262'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"llm-d-routing-sidecar\" in pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz\" is waiting to start: PodInitializing","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:204 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '8398cd00-190b-4c66-b9ec-3d4168523e37', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Mon, 13 Jul 2026 19:47:55 GMT', 'Content-Length': '245'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz\" is waiting to start: PodInitializing","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:204 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '9532de9d-0908-457d-a9e8-1e6f04eb1e62', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Mon, 13 Jul 2026 19:47:55 GMT', 'Content-Length': '247'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt\" is waiting to start: ContainerCreating","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 {"level":"info","ts":1783971203.9109828,"logger":"setup","caller":"runner/runner.go:196","msg":"GIE build","commit-sha":"181aa8358916e19b8844ccc752b2d6153d4b2ad6","build-ref":"v0.9.0-rc.2"} [e2e-llm-inference-service] Flag --model-server-metrics-scheme has been deprecated, This flag is deprecated. Configure via EndpointPickerConfig data layer plugin parameters instead. [e2e-llm-inference-service] {"level":"info","ts":1783971203.9111593,"logger":"setup","caller":"runner/runner.go:217","msg":"Flags processed","flags":{"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n- type: queue-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n- parameters:\n deciders:\n prefill: always-disagg-pd-decider\n type: disagg-profile-handler\nschedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"https","pool-group":"inference.networking.k8s.io","pool-name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","pool-namespace":"kserve-ci-e2e-test","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9112742,"logger":"setup.trace","caller":"tracing/telemetry.go:123","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9121778,"caller":"loader/configloader.go:89","msg":"DEPRECATION: apiVersion inference.networking.x-k8s.io/v1alpha1/EndpointPickerConfig is deprecated","replacement":"llm-d.ai/v1alpha1/EndpointPickerConfig"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9122477,"caller":"loader/configloader.go:121","msg":"Loaded raw configuration","config":"{Plugins: [{Type: disagg-headers-handler} {Type: prefill-filter} {Type: decode-filter} {Type: queue-scorer} {Type: prefix-cache-scorer} {Type: max-score-picker} {Type: always-disagg-pd-decider} {Type: disagg-profile-handler, Parameters: {\"deciders\":{\"prefill\":\"always-disagg-pd-decider\"}}}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}]}"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.912261,"logger":"setup","caller":"runner/runner.go:622","msg":"Data layer: ENABLED"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9127414,"logger":"setup","caller":"runner/runner.go:281","msg":"Raw config after phase one","config":{"apiVersion":"inference.networking.x-k8s.io/v1alpha1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"disagg-headers-handler","parameters":null,"type":"disagg-headers-handler"},{"name":"prefill-filter","parameters":null,"type":"prefill-filter"},{"name":"decode-filter","parameters":null,"type":"decode-filter"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"},{"name":"always-disagg-pd-decider","parameters":null,"type":"always-disagg-pd-decider"},{"name":"disagg-profile-handler","parameters":{"deciders":{"prefill":"always-disagg-pd-decider"}},"type":"disagg-profile-handler"}],"schedulingProfiles":[{"name":"prefill","plugins":[{"pluginRef":"prefill-filter","weight":null},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]},{"name":"decode","plugins":[{"pluginRef":"decode-filter","weight":null},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]}]}} [e2e-llm-inference-service] {"level":"info","ts":1783971203.940317,"caller":"disagg/disagg_profile_handler.go:186","msg":"No deciders.encode configured, E disaggregation disabled"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9404082,"logger":"utilization-detector/utilization-detector","caller":"utilization/detector.go:83","msg":"Creating new UtilizationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms","headroom":0} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9405258,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"vllm","mapping":"Mapping{all specs enabled}"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9405854,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"sglang","mapping":"Mapping{disabled: [lora]}"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9406507,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"trtllm-serve","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.940753,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton-tensorrt-llm","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.940806,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton","mapping":"Mapping{disabled: [kv, lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9409168,"caller":"loader/configloader.go:154","msg":"Instantiated all plugins and applied system defaults. Effective raw configuration","config":"{Plugins: [{Name: disagg-headers-handler, Type: disagg-headers-handler} {Name: prefill-filter, Type: prefill-filter} {Name: decode-filter, Type: decode-filter} {Name: queue-scorer, Type: queue-scorer} {Name: prefix-cache-scorer, Type: prefix-cache-scorer} {Name: max-score-picker, Type: max-score-picker} {Name: always-disagg-pd-decider, Type: always-disagg-pd-decider} {Name: disagg-profile-handler, Type: disagg-profile-handler, Parameters: {\"deciders\":{\"prefill\":\"always-disagg-pd-decider\"}}} {Name: fcfs-ordering-policy, Type: fcfs-ordering-policy} {Name: global-strict-fairness-policy, Type: global-strict-fairness-policy} {Name: static-usage-limit-policy, Type: static-usage-limit-policy} {Name: openai-parser, Type: openai-parser} {Name: anthropic-parser, Type: anthropic-parser} {Name: vllmhttp-parser, Type: vllmhttp-parser} {Name: utilization-detector, Type: utilization-detector} {Name: metrics-data-source, Type: metrics-data-source} {Name: core-metrics-extractor, Type: core-metrics-extractor}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}], DataLayer: {Sources: [{PluginRef: metrics-data-source, Extractors: [{PluginRef: core-metrics-extractor}]}], Discovery: }, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}]}}"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9410353,"caller":"approximateprefix/plugin.go:88","msg":"Prefix DataProducer initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":2048,"maxPrefixTokensToMatch":131072,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9411244,"caller":"approximateprefix/plugin.go:111","msg":"WARNING: configured blockSizeTokens is below the recommended minimum, overriding it.","blockSizeTokens":16,"minimum":64,"issue":"https://github.com/llm-d/llm-d-router/issues/1158"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9411564,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"approx-prefix-cache-producer/approx-prefix-cache-producer","dataKey":"PrefixCacheMatchInfoDataKey/approx-prefix-cache-producer","consumer":"disagg-profile-handler"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.941183,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"token-producer/token-producer","dataKey":"TokenizedPrompt/token-producer","consumer":"disagg-profile-handler"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9413102,"caller":"runner/runner.go:685","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9413216,"logger":"setup","caller":"runner/runner.go:308","msg":"EPP config after phase two","config":"{SchedulerConfig:{ProfileHandler: disagg-profile-handler/disagg-profile-handler, Profiles: map[decode:{Filters: [decode-filter/by-label], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker} prefill:{Filters: [prefill-filter/by-label], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker}]} SaturationDetector:0xc0007df140 DataConfig:{Sources:[{Plugin:0xc00043d440 Extractors:[0xc0007df380]}]} FlowControlConfig: ParserRegistry:0xc0007df980}"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.965653,"logger":"setup","caller":"runner/runner.go:352","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.965692,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9657078,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9657135,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9657183,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9657226,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/cmdline"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9657273,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/profile"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9657316,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/symbol"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9657357,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/trace"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9657404,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9657452,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9657493,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9657564,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/plugins/state"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.965761,"logger":"setup","caller":"runner/runner.go:373","msg":"parsed config","scheduler-config":"{ProfileHandler: disagg-profile-handler/disagg-profile-handler, Profiles: map[decode:{Filters: [decode-filter/by-label], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker} prefill:{Filters: [prefill-filter/by-label], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9658146,"logger":"setup","caller":"datalayer/runtime.go:99","msg":"Configuring datalayer runtime","numSources":1} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9658234,"logger":"setup","caller":"datalayer/runtime.go:118","msg":"Processing source","source":"metrics-data-source","numExtractors":1} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9658375,"logger":"setup","caller":"datalayer/runtime.go:147","msg":"Source configured","source":"metrics-data-source","extractors":["core-metrics-extractor/core-metrics-extractor"]} [e2e-llm-inference-service] {"level":"info","ts":1783971203.965851,"logger":"setup","caller":"datalayer/runtime.go:206","msg":"Datalayer runtime configured","pollers":1,"notifiers":0,"endpointSources":0} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9658608,"logger":"setup","caller":"runner/runner.go:833","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9659488,"logger":"setup","caller":"runner/runner.go:721","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":1783971203.965962,"logger":"setup","caller":"runner/runner.go:260","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9659882,"logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9662018,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.966292,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9663887,"logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":1783971203.966552,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9665587,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9666724,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9666936,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.967517,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9676504,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9719832,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9721262,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.972166,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783971203.9752457,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783971204.067052,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1783971204.0670915,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783971204.0684316,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1783971204.068512,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783971204.1688225,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":1783971204.1688638,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783971204.1688201,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1783971204.1688943,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783971204.169023,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","reconcileID":"ac8bbd72-fbcd-4481-834a-830b376205de"} [e2e-llm-inference-service] {"level":"info","ts":1783971240.1993687,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","reconcileID":"d291df78-fad8-434f-8122-a8d9977775ff"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 6744f490-76bf-4c6f-b852-7804d6c3bda3 [e2e-llm-inference-service] resourceVersion: '48430' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-13T19:33:56Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:56Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.132.0.52 [e2e-llm-inference-service] nodeName: ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb [e2e-llm-inference-service] uid: dbcef857-215f-4fa4-8519-e5029d249889 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: f3b24567-fdf0-459f-83ff-71d76db14f90 [e2e-llm-inference-service] resourceVersion: '47704' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:22Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-13T19:33:22Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:22Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 762ad0f9-19a6-4a36-b4eb-a50f7291ab13 [e2e-llm-inference-service] resourceVersion: '47676' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:21Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 7db79954c [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.57/23"],"mac_address":"0a:58:0a:86:00:39","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.57/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c [e2e-llm-inference-service] uid: 1749d0a4-ca3f-4148-9058-a8de27873038 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-128-215 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"1749d0a4-ca3f-4148-9058-a8de27873038"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-6x8dq [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --decoder-use-tls=true [e2e-llm-inference-service] - --prefiller-use-tls=true [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-6x8dq [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"facebook/opt-125m\"\ [e2e-llm-inference-service] \ \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\" \\\n --port 8001\ [e2e-llm-inference-service] \ \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n --enable-ssl-refresh\ [e2e-llm-inference-service] \ \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-6x8dq [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] nodeName: ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-gv2sm [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-68275 [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:33:21Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:33:21Z' [e2e-llm-inference-service] reason: ContainersNotInitialized [e2e-llm-inference-service] message: 'containers with incomplete status: [llm-d-routing-sidecar]' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:33:21Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [llm-d-routing-sidecar main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:33:21Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [llm-d-routing-sidecar main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:33:21Z' [e2e-llm-inference-service] hostIP: 10.0.128.215 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.128.215 [e2e-llm-inference-service] startTime: '2026-07-13T19:33:21Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: PodInitializing [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-6x8dq [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: PodInitializing [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-6x8dq [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79497db4cc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 74e7b5d4-9d6e-4736-98c3-09d48ed6dec2 [e2e-llm-inference-service] resourceVersion: '47694' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:22Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 79497db4cc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.58/23"],"mac_address":"0a:58:0a:86:00:3a","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.58/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79497db4cc [e2e-llm-inference-service] uid: cb14e08d-12e1-4aae-a76a-6250887b376c [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-128-215 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:22Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:22Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cb14e08d-12e1-4aae-a76a-6250887b376c"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:22Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-njklf [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-njklf [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-gv2sm [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:33:22Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:33:22Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:33:22Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:33:22Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:33:22Z' [e2e-llm-inference-service] hostIP: 10.0.128.215 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.128.215 [e2e-llm-inference-service] startTime: '2026-07-13T19:33:22Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: ContainerCreating [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-njklf [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb [e2e-llm-inference-service] generateName: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-67b4bb9646- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: dbcef857-215f-4fa4-8519-e5029d249889 [e2e-llm-inference-service] resourceVersion: '48429' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 67b4bb9646 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.52/23"],"mac_address":"0a:58:0a:84:00:34","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.52/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.132.0.52\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:84:00:34\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-67b4bb9646 [e2e-llm-inference-service] uid: 2117c812-297e-48f5-ab2f-fe0a2139d666 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-131-203 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2117c812-297e-48f5-ab2f-fe0a2139d666"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:56Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.132.0.52"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-657d7 [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n\ [e2e-llm-inference-service] - type: queue-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] - type: always-disagg-pd-decider\n- parameters:\n deciders:\n prefill:\ [e2e-llm-inference-service] \ always-disagg-pd-decider\n type: disagg-profile-handler\nschedulingProfiles:\n\ [e2e-llm-inference-service] - name: prefill\n plugins:\n - pluginRef: prefill-filter\n - pluginRef: queue-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef:\ [e2e-llm-inference-service] \ max-score-picker\n- name: decode\n plugins:\n - pluginRef: decode-filter\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-657d7 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-gv2sm [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-x8klk [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:33:24Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:33:56Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:33:56Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] hostIP: 10.0.131.203 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.131.203 [e2e-llm-inference-service] podIP: 10.132.0.52 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.132.0.52 [e2e-llm-inference-service] startTime: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-router-endpoint-picker@sha256:06b6c75d77afd0e07053402752a9736c2dfbc12a306d0d37d963aac4c1d4e6a6 [e2e-llm-inference-service] containerID: cri-o://bcc9612595c698fba65a2883f5120d0c36c06f7f62c6dd9d9e80efc8f6b0e768 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-657d7 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 866f8078-da33-44cf-9749-fdef0433cb29 [e2e-llm-inference-service] resourceVersion: '47712' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:22Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-x8klk [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:22Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-x8klk"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:22Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"24426e0c-aa4a-4fe5-8006-7fec4a21e2b0"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-gv2sm"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-gv2sm [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-x8klk [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-gv2sm [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-x8klk [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 03697c6e-6111-4119-9c9c-67347cc07d91 [e2e-llm-inference-service] resourceVersion: '47642' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:21Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-68275 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-68275"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"24426e0c-aa4a-4fe5-8006-7fec4a21e2b0"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-gv2sm"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-gv2sm [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-68275 [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-gv2sm [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-68275 [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 5c4d536b-1874-4ec5-ab60-7e47c762e457 [e2e-llm-inference-service] resourceVersion: '47740' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"24426e0c-aa4a-4fe5-8006-7fec4a21e2b0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.65.73 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.65.73 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 6d456b87-f41f-47b6-8494-cdcc73b79426 [e2e-llm-inference-service] resourceVersion: '47702' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:22Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:22Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"24426e0c-aa4a-4fe5-8006-7fec4a21e2b0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.94.122 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.94.122 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 1e959628-0a9a-4d27-9513-a0efaf736a19 [e2e-llm-inference-service] resourceVersion: '54970' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:21Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"24426e0c-aa4a-4fe5-8006-7fec4a21e2b0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-13T19:43:22Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:unavailableReplicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --decoder-use-tls=true [e2e-llm-inference-service] - --prefiller-use-tls=true [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] unavailableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-13T19:33:21Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:33:21Z' [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-13T19:43:22Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:43:22Z' [e2e-llm-inference-service] reason: ProgressDeadlineExceeded [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c" [e2e-llm-inference-service] has timed out progressing. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 28309417-1a7b-4b62-9a08-8f4891b9291e [e2e-llm-inference-service] resourceVersion: '54978' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:22Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:22Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"24426e0c-aa4a-4fe5-8006-7fec4a21e2b0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-13T19:43:23Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:unavailableReplicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n\ [e2e-llm-inference-service] \ ${SHUTDOWN_TIMEOUT_ARGS} \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] unavailableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-13T19:33:22Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:33:22Z' [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-13T19:43:23Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:43:23Z' [e2e-llm-inference-service] reason: ProgressDeadlineExceeded [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79497db4cc" [e2e-llm-inference-service] has timed out progressing. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 51f47b6f-9557-4e0e-98b2-65f1fbd9dbac [e2e-llm-inference-service] resourceVersion: '48433' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"24426e0c-aa4a-4fe5-8006-7fec4a21e2b0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:56Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type:\ [e2e-llm-inference-service] \ decode-filter\n- type: queue-scorer\n- type: prefix-cache-scorer\n- type:\ [e2e-llm-inference-service] \ max-score-picker\n- type: always-disagg-pd-decider\n- parameters:\n \ [e2e-llm-inference-service] \ deciders:\n prefill: always-disagg-pd-decider\n type: disagg-profile-handler\n\ [e2e-llm-inference-service] schedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: decode-filter\n - pluginRef: queue-scorer\n weight:\ [e2e-llm-inference-service] \ 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-13T19:33:56Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:33:56Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-13T19:33:56Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-67b4bb9646" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 1749d0a4-ca3f-4148-9058-a8de27873038 [e2e-llm-inference-service] resourceVersion: '47673' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:21Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 7db79954c [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] uid: 1e959628-0a9a-4d27-9513-a0efaf736a19 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"1e959628-0a9a-4d27-9513-a0efaf736a19"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 7db79954c [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 7db79954c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --decoder-use-tls=true [e2e-llm-inference-service] - --prefiller-use-tls=true [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79497db4cc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: cb14e08d-12e1-4aae-a76a-6250887b376c [e2e-llm-inference-service] resourceVersion: '47692' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:22Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 79497db4cc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] uid: 28309417-1a7b-4b62-9a08-8f4891b9291e [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:22Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"28309417-1a7b-4b62-9a08-8f4891b9291e"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:22Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 79497db4cc [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 79497db4cc [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n\ [e2e-llm-inference-service] \ ${SHUTDOWN_TIMEOUT_ARGS} \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-67b4bb9646 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 2117c812-297e-48f5-ab2f-fe0a2139d666 [e2e-llm-inference-service] resourceVersion: '48432' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 67b4bb9646 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler [e2e-llm-inference-service] uid: 51f47b6f-9557-4e0e-98b2-65f1fbd9dbac [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"51f47b6f-9557-4e0e-98b2-65f1fbd9dbac"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:56Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 67b4bb9646 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 67b4bb9646 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type:\ [e2e-llm-inference-service] \ decode-filter\n- type: queue-scorer\n- type: prefix-cache-scorer\n- type:\ [e2e-llm-inference-service] \ max-score-picker\n- type: always-disagg-pd-decider\n- parameters:\n \ [e2e-llm-inference-service] \ deciders:\n prefill: always-disagg-pd-decider\n type: disagg-profile-handler\n\ [e2e-llm-inference-service] schedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: decode-filter\n - pluginRef: queue-scorer\n weight:\ [e2e-llm-inference-service] \ 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: c165f6c8-c486-4b7c-a2c7-ce92c862994e [e2e-llm-inference-service] resourceVersion: '47731' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"24426e0c-aa4a-4fe5-8006-7fec4a21e2b0"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 4ad2b9cd-d559-4669-8f43-f20403e85187 [e2e-llm-inference-service] resourceVersion: '47654' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:21Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"24426e0c-aa4a-4fe5-8006-7fec4a21e2b0"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 890c1e67-9d6c-453f-8232-eb14592166fe [e2e-llm-inference-service] resourceVersion: '47729' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"24426e0c-aa4a-4fe5-8006-7fec4a21e2b0"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 97ea1246-2048-4617-bb62-0c03389e8d65 [e2e-llm-inference-service] resourceVersion: '47651' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:21Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"24426e0c-aa4a-4fe5-8006-7fec4a21e2b0"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service-mxkpt [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-epp-service- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 1cb69f8e-c5e1-4ae5-a532-b335ed127ebd [e2e-llm-inference-service] resourceVersion: '48431' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-13T19:33:56Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] uid: 5c4d536b-1874-4ec5-ab60-7e47c762e457 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:56Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5c4d536b-1874-4ec5-ab60-7e47c762e457"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.132.0.52 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb [e2e-llm-inference-service] uid: dbcef857-215f-4fa4-8519-e5029d249889 [e2e-llm-inference-service] nodeName: ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-shznqf [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: e721d3a1-cd2a-419a-bcde-8b446b609567 [e2e-llm-inference-service] resourceVersion: '47705' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:22Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-13T19:33:22Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] uid: 6d456b87-f41f-47b6-8494-cdcc73b79426 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:22Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6d456b87-f41f-47b6-8494-cdcc73b79426"}: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: null [e2e-llm-inference-service] ports: null [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: c165f6c8-c486-4b7c-a2c7-ce92c862994e [e2e-llm-inference-service] resourceVersion: '47731' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"24426e0c-aa4a-4fe5-8006-7fec4a21e2b0"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 4ad2b9cd-d559-4669-8f43-f20403e85187 [e2e-llm-inference-service] resourceVersion: '47654' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:21Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"24426e0c-aa4a-4fe5-8006-7fec4a21e2b0"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 890c1e67-9d6c-453f-8232-eb14592166fe [e2e-llm-inference-service] resourceVersion: '47729' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"24426e0c-aa4a-4fe5-8006-7fec4a21e2b0"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 97ea1246-2048-4617-bb62-0c03389e8d65 [e2e-llm-inference-service] resourceVersion: '47651' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:21Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-13T19:33:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"24426e0c-aa4a-4fe5-8006-7fec4a21e2b0"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"24426e0c-aa4a-4fe5-8006-7fec4a21e2b0"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:34:00Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-13T19:34:00Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-13T19:34:00Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] resourceVersion: '48495' [e2e-llm-inference-service] uid: 0e52916c-24dc-4315-8750-44cf928b9e02 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:34:00Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:33:24Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"24426e0c-aa4a-4fe5-8006-7fec4a21e2b0"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:34:00Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-13T19:34:00Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-13T19:34:00Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] resourceVersion: '48495' [e2e-llm-inference-service] uid: 0e52916c-24dc-4315-8750-44cf928b9e02 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:34:00Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:33:24Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"24426e0c-aa4a-4fe5-8006-7fec4a21e2b0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-13T19:34:00Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] resourceVersion: '48477' [e2e-llm-inference-service] uid: c4e5e957-3f2e-442f-bb63-21e0e8bfedf8 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:34:00Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:34:00Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:24Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"24426e0c-aa4a-4fe5-8006-7fec4a21e2b0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:33:24Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-13T19:33:27Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] resourceVersion: '47947' [e2e-llm-inference-service] uid: 5f619c06-021c-4e8e-989a-78118597a393 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:33:26Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:33:27Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:24Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"24426e0c-aa4a-4fe5-8006-7fec4a21e2b0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:33:24Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] resourceVersion: '47825' [e2e-llm-inference-service] uid: 1857fd15-f859-4279-a030-d4b491b40189 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:34:00Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"24426e0c-aa4a-4fe5-8006-7fec4a21e2b0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:34:00Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] resourceVersion: '48483' [e2e-llm-inference-service] uid: 82710477-6a41-4a05-8023-140ffd0598e0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:24Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"24426e0c-aa4a-4fe5-8006-7fec4a21e2b0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:33:24Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] resourceVersion: '47842' [e2e-llm-inference-service] uid: d6818870-8671-4a27-91c5-4cb91745d9e8 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:24Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"24426e0c-aa4a-4fe5-8006-7fec4a21e2b0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:33:24Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] resourceVersion: '47825' [e2e-llm-inference-service] uid: 1857fd15-f859-4279-a030-d4b491b40189 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:34:00Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"24426e0c-aa4a-4fe5-8006-7fec4a21e2b0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:34:00Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] resourceVersion: '48483' [e2e-llm-inference-service] uid: 82710477-6a41-4a05-8023-140ffd0598e0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:24Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"24426e0c-aa4a-4fe5-8006-7fec4a21e2b0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:33:24Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] resourceVersion: '47842' [e2e-llm-inference-service] uid: d6818870-8671-4a27-91c5-4cb91745d9e8 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:24Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"24426e0c-aa4a-4fe5-8006-7fec4a21e2b0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:33:24Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] resourceVersion: '47825' [e2e-llm-inference-service] uid: 1857fd15-f859-4279-a030-d4b491b40189 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:34:00Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"24426e0c-aa4a-4fe5-8006-7fec4a21e2b0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:34:00Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] resourceVersion: '48483' [e2e-llm-inference-service] uid: 82710477-6a41-4a05-8023-140ffd0598e0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:24Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"24426e0c-aa4a-4fe5-8006-7fec4a21e2b0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:33:24Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] resourceVersion: '47842' [e2e-llm-inference-service] uid: d6818870-8671-4a27-91c5-4cb91745d9e8 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"24426e0c-aa4a-4fe5-8006-7fec4a21e2b0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:33:23Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 24426e0c-aa4a-4fe5-8006-7fec4a21e2b0 [e2e-llm-inference-service] resourceVersion: '47782' [e2e-llm-inference-service] uid: c6d2d700-e4ea-40f7-baf3-a97f7d800502 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:47:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 67b4bb9646 [e2e-llm-inference-service] timestamp: '2026-07-13T19:47:41Z' [e2e-llm-inference-service] window: 19.377s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 952108n [e2e-llm-inference-service] memory: 20812Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-07-13T19:47:56.130887] end - ❌ 902.670s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-13T19:34:01Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-13T19:34:01Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-13T19:34:01Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-13T19:34:01Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-13T19:34:01Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:33:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] _ test_llm_inference_service[router-managed-workload-simulated-dp-ep-cpu-model-pvc] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-simulated-dp-ep-cpu', 'model-pvc'], prompt='KServe is a', service_name... {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[0], ROUTER_ROUTES[1]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[1], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[1]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[2], ROUTER_ROUTES[3]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[create_scheduler_configmap], [e2e-llm-inference-service] after_test=[delete_scheduler_configmap], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Precise prefix KV cache routing test [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="precise-prefix-cache-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name=f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] [e2e-llm-inference-service] test_case.llm_service.metadata.annotations[ [e2e-llm-inference-service] "security.opendatahub.io/enable-auth" [e2e-llm-inference-service] ] = "false" [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:812: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...pu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-13T19:47:57.037603', start_time = 1783972077.0378778 [e2e-llm-inference-service] duration = 900.3918588161469, timestamp_end = '2026-07-13T20:02:57.429741' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....p-ep-cpu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1204: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f604d03d6c0> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1215: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-13T19:48:59Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-13T19:48:59Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-13T19:48:24Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-13T19:48:59Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-13T19:48:59Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-13T19:48:59Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:48:24Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:48:24Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1199: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1752 PVC e2e-pvc-model-storage already exists in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1962 Model download Job already completed, skipping [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig router-managed-llmisvc-model-pv-2f105a64 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-model-pv-2f105a64 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-model-pv-2f105a64 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig workload-simulated-dp-ep-cpu-ll-699c687c in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig workload-simulated-dp-ep-cpu-ll-699c687c [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig workload-simulated-dp-ep-cpu-ll-699c687c [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-cfc8d654 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-cfc8d654 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-cfc8d654 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-07-13T19:47:56.761716] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-simulated-dp-ep-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-model-pvc-router-manage-2577e794', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-2577e794', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-2f105a64'}, [e2e-llm-inference-service] {'name': 'workload-simulated-dp-ep-cpu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-13T19:47:56.774337] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-2577e794', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-2f105a64'}, [e2e-llm-inference-service] {'name': 'workload-simulated-dp-ep-cpu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-13T19:47:57.037499] end - ✅ in 0.263s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-13T19:47:57.037603] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-2577e794', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-2f105a64'}, [e2e-llm-inference-service] {'name': 'workload-simulated-dp-ep-cpu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-13T19:48:24Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-13T19:48:24Z', 'message': 'Inference Pool kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-13T19:48:24Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-13T19:48:24Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-13T19:48:24Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-13T19:48:24Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:48:24Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:48:24Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-13T19:48:59Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-13T19:48:59Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-13T19:48:24Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-13T19:48:59Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-13T19:48:59Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-13T19:48:59Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:48:24Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:48:24Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1219 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-13T19:48:59Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-13T19:48:59Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-13T19:48:24Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-13T19:48:59Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-13T19:48:59Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-13T19:48:59Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:48:24Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:48:24Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-13T20:02:57.429741] end - ❌ 900.392s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-13T19:48:59Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-13T19:48:59Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-13T19:48:24Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-13T19:48:59Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-13T19:48:59Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-13T19:48:59Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:48:24Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:48:24Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:831 [router-managed-workload-simulated-dp-ep-cpu-model-pvc] ❌ ERROR: Failed to call llm inference service llmisvc-model-pvc-router-manage-2577e794: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-13T19:48:59Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-13T19:48:59Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-13T19:48:24Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-13T19:48:59Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-13T19:48:59Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-13T19:48:59Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:48:24Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:48:24Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1240 🔍 # Diagnostics for 'llmisvc-model-pvc-router-manage-2577e794' in 'kserve-ci-e2e-test' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1241 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1242 # LLMInferenceService llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1245 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:47:56Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:47:56Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:48:16Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-13T19:48:59Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] resourceVersion: '58591' [e2e-llm-inference-service] uid: d209db60-7b81-4f60-9cb9-be0ce92a99c3 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-llmisvc-model-pv-2f105a64 [e2e-llm-inference-service] - name: workload-simulated-dp-ep-cpu-ll-699c687c [e2e-llm-inference-service] - name: model-pvc-llmisvc-model-pvc-rou-cfc8d654 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://a7407f728462246e2be4fe1c58f85714-238314375.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a7407f728462246e2be4fe1c58f85714-238314375.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:48:59Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:48:59Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:48:24Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:48:59Z' [e2e-llm-inference-service] message: LWS is progressing [e2e-llm-inference-service] reason: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:48:59Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:48:59Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:48:24Z' [e2e-llm-inference-service] message: LWS is progressing [e2e-llm-inference-service] reason: Progressing [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:48:24Z' [e2e-llm-inference-service] message: LWS is progressing [e2e-llm-inference-service] reason: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://a7407f728462246e2be4fe1c58f85714-238314375.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:44 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:45 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9 to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:45 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:45 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:45 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:31 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.45:8000/health": dial tcp 10.134.0.45:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:47 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:02 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.45:8000/health": dial tcp 10.134.0.45:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-ccd54d6d4-cgwk9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:42 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:42 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:42 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:43 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:43 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:43 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:47 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:47 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-router-scheduler-58f5ddbc8f-qgm5g [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-router-scheduler-58f5ddbc8f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-ccd54d6d4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:36 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy auth-disabled-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "auth-disabled-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-disabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-disabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-disabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-disabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:42 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-disabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:08 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:08 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:08 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:08 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:11 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulling Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:38 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 27.071s (27.071s including waiting). Image size: 3531177328 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:38 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:38 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:18 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.41:8000/health": dial tcp 10.134.0.41:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:49 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.41:8000/health": context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-7dfddf4646-c84nz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:08 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:08 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:08 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:08 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:09 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:11 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.571s (1.571s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:11 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:11 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-7dfddf4646 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-enabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-enabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-enabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-enabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-enabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:28 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-enabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:07 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:07 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:07 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:53 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.43:8000/health": dial tcp 10.134.0.43:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:05 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:20 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.43:8000/health": dial tcp 10.134.0.43:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-58b5cf756b-27d2s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:04 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:04 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:04 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:04 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:05 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:05 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:05 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:05 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:05 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-router-scheduler-6c486c5958 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-58b5cf756b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-invalid-token-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-invalid-token-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-invalid-token-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-invalid-token-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:08 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-invalid-token-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:04 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-invalid-token-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.55/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:36 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:36 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:36 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:32:01 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.55:8001/health": dial tcp 10.134.0.55:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:32:59 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:32:59 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-6d9d59446f-spptn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.56/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:35 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:35 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:35 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:32:01 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.56:8000/health": dial tcp 10.134.0.56:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:32:59 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcfmn6z to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:33 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:33 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:33 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:32:59 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:32:59 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-router-scheduler-5f5fcc46c8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-6d9d59446f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:20 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-pd-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-pd-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:29:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:32:36 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-pd-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:32:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.51/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:36 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:36 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:36 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:21:21 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.51:8000/health": dial tcp 10.134.0.51:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:21:45 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-965ccd984-nft7b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-685c7787ffkr6 to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:32 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:32 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:32 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:33 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:33 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:33 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:21:45 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:21:45 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-router-scheduler-685c77879f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-965ccd984 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:30 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:20:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:20:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:20:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:21:36 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:21:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/e2e-pvc-model-download-w2mvs to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:35 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-3823dadc-7c52-412c-8ef9-84caab00a904" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:39 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:39 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:39 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:39 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:29 kserve-ci-e2e-test job-controller Normal SuccessfulCreate Created pod: e2e-pvc-model-download-w2mvs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:48 kserve-ci-e2e-test job-controller Normal Completed Job completed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:29 kserve-ci-e2e-test persistentvolume-controller Normal WaitForFirstConsumer waiting for first consumer to be created before binding [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:29 kserve-ci-e2e-test persistentvolume-controller Normal ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:29 kserve-ci-e2e-test ebs.csi.aws.com_aws-ebs-csi-driver-controller-d8db6f98-dhbrp_42e66c8b-a583-47f3-9a2d-f39fd5be165e Normal Provisioning External provisioner is provisioning volume for claim "kserve-ci-e2e-test/e2e-pvc-model-storage" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:32 kserve-ci-e2e-test ebs.csi.aws.com_aws-ebs-csi-driver-controller-d8db6f98-dhbrp_42e66c8b-a583-47f3-9a2d-f39fd5be165e Normal ProvisioningSucceeded Successfully provisioned volume pvc-3823dadc-7c52-412c-8ef9-84caab00a904 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:02 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5cktt6 to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:03 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:03 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:04 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.206s (1.206s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:04 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:04 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:16 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:02 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:20 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:20 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:21 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:21 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:21 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:50 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:05 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.40:8000/health": EOF [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:20 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:52 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:53 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.61/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:53 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:53 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:53 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:57 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:57 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:57 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:39:43 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.61:8000/health": dial tcp 10.134.0.61:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:40:46 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:02 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.61:8000/health": dial tcp 10.134.0.61:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:52 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:51 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-3c960099-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-3c960099-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:38:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:40:24 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-3c960099] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6 to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.53/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:03 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" in 778ms (778ms including waiting). Image size: 44914394 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:03 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:03 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:04 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:04 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:04 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:08 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:08 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:08 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:28:12 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.53:8001/health": dial tcp 10.134.0.53:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:28:49 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:28:49 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9 to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.54/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:06 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:06 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:06 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:28:32 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.54:8000/health": dial tcp 10.134.0.54:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:28:49 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f89477b49 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:01 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:28:43 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-50bc673d] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:28:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27 to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:00 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:00 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:00 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:18:46 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.49:8000/health": dial tcp 10.134.0.49:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:07 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:22 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.49:8000/health": dial tcp 10.134.0.49:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:54 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:18:57 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-87882a8e] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.55/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:31 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:31 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:31 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:35 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulling Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:42:02 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 27.76s (27.76s including waiting). Image size: 3531177328 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:42:03 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:42:03 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:49:37 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 in StatefulSet llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.62/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:31 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:31 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:34 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:34 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:34 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:43:00 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.62:8000/health": dial tcp 10.134.0.62:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:49:21 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:49:36 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.62:8000/health": dial tcp 10.134.0.62:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test leaderworkerset Normal CreatingRevision Creating revision with key 848c75bc96 for a newly created LeaderWorkerSet [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created leader statefulset llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Replicas are progressing, with 0 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 in StatefulSet llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created worker statefulset for leader pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:43:10 kserve-ci-e2e-test leaderworkerset Normal AllGroupsReady All replicas are ready, with 1 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:15 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-scc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.LeaderWorkerSet kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test386bd5808c5c450f11fd8632e1f821fb-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-dc21cb14] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:49:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test386bd5808c5c450f11fd8632e1f821fb-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:15 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4ctbbm to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:16 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:16 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:16 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:31 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:15 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:13 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:26 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-e95b1dc1] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:24 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:24 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:24 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:39 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.47:8000/health": dial tcp 10.134.0.47:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:17 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:18 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler-6698955b74 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher2z97 to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:20 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:20 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:20 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:20 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:20 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:17 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:17 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:18 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:18 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:33 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:33 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:50 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-7ca60146] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:52 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:53 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:53 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:58 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.48:8000/health": dial tcp 10.134.0.48:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:10 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-844f8ff647 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schelnjhn to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:49 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:49 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:49 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:49 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:50 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:50 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:50 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:09 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:09 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:47 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:15:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:08 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-ba4d693a] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:48:21 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 in StatefulSet llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:48:21 kserve-ci-e2e-test leaderworkerset Normal CreatingRevision Creating revision with key 686d468674 for a newly created LeaderWorkerSet [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:48:21 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created leader statefulset llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:48:21 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Replicas are progressing, with 0 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:48:21 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 in StatefulSet llmisvc-model-pvc-router-manage-2577e794-kserve-mn successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:48:21 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created worker statefulset for leader pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:48:17 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-2577e794-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:48:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:48:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-scc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:48:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:48:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.LeaderWorkerSet kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:48:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:48:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:48:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test37591b20e96e9663d45a730d03070f1e-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:48:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:48:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:48:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:58:33 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:48:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:57 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-3823dadc-7c52-412c-8ef9-84caab00a904" Volume is already used by pod(s) llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-59b9d263-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-59b9d263-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testc9569cf4801efc0ed27b2f25ffaee875-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:18:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:58:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:18:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:21 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:22 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:22 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79497db4cc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:21 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:03 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-e8706282-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testf5d060a5eb39a04e074b78907a1556a6-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:34:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:58:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:34:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:53:44 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4hjlzp to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:53:44 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:53:44 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:53:44 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:53:44 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:55:53 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:53:44 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:53:38 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:53:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:53:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:53:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:53:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:53:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test80f99418f0f3eb4c80f02e6144a6d4f2-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:53:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:53:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:53:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:53:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:54:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:54:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:54:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:54:30 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-4c1ba212] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:54:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test80f99418f0f3eb4c80f02e6144a6d4f2-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fb2h9x to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:32 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:48 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.38:8000/health": dial tcp 10.133.0.38:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:46 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:31 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-4f8c0978] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddcrqxt to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:42 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:42 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:42 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:15 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:39 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:14 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-a50492e9] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:52 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:13:07 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.38:8000/health": dial tcp 10.132.0.38:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-fwwrx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:06 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-4b931143-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-4b931143-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:40 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-4b931143] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:01 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jt7jj to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:35 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:01 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:58 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-5b1e8f15-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:34 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-5b1e8f15] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879ntk8x to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:33 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:48 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.35:8000/health": dial tcp 10.132.0.35:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:57 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-e45d1f79-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-e45d1f79-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:32 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-e45d1f79] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:56 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:56 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:33 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:33 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:11:55 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler-6fcb489785 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:48:21 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:48:21 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-7dbcb75dbc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2 to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:48:22 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:48:22 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:48:22 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:48:22 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheduler-6b4848c4bd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schefs4wn to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.54/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:30 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:31 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:31 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:32 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:32 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:41:32 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:49:21 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:49:21 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4 to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:53:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.58/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:53:45 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:53:45 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:53:45 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:55:53 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:55:53 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:53:44 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche6fld4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:53:44 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheduler-79866dc455 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:02 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheduler-69d8c68fd8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schenm4bp to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:03 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:03 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:03 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:03 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:04 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:04 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:26:04 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:28:49 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:28:49 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-5dd88bfbb7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel7mgq to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:58 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:00 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.205s (1.205s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:00 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:00 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:33 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:07:33 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:52 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:52 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler-5f555d4d85 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:52 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:52 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:52 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:52 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:17:57 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-3823dadc-7c52-412c-8ef9-84caab00a904" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:56 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler-688ff6d544 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherchj5 to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:57 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:57 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:57 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:57 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:57 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:57 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:16:57 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:07 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:19:07 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:42 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:42 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:42 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:15 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:15 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7bbzq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:41 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler-6d86bd4d9d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:23 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:23 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-67b4bb9646 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:23 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-3823dadc-7c52-412c-8ef9-84caab00a904" Volume is already used by pod(s) llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:23 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.52/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:23 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:23 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:23 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-68cc9685d6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schex5jnk to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:08 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:08 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:08 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:10:08 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:52 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:12:52 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-749449dbc8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek22wf to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:08:02 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:35 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:09:35 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2 to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:18 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:19 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.48s (1.48s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:19 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:19 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:26 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-v494f to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" in 5.156s (5.156s including waiting). Image size: 300570036 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:23 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:25 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.796s (1.796s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:25 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:25 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:26 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:13 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-6ffc955574-v494f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:13 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-6ffc955574-p26g2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-d84595fbj5ftw to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" in 4.75s (4.75s including waiting). Image size: 300570036 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:20 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:22 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.694s (1.694s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:52 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:52 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:22 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:52 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" in 30.148s (30.148s including waiting). Image size: 2989890188 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:52 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:52 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:52 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:05 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning Unhealthy Liveness probe failed: timeout: failed to connect service "10.132.0.34:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:26 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:26 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:26 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-d84595fb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:13 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-6ffc955574 from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:07 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:25 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [precise-prefix-cache-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:06:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kgtbh to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:23 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:23 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:25 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" in 2.49s (2.49s including waiting). Image size: 181251556 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:26 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:26 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:32 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.133.0.33:15021/healthz/ready": dial tcp 10.133.0.33:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:26 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:08 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-lfprg to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:56 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:56 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:59 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" in 3.513s (3.513s including waiting). Image size: 181251556 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:59 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:59 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:06 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-r2bjb to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:09 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:20 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:20 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:20 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.46:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.46:15021/healthz/ready": read tcp 10.132.0.2:35418->10.132.0.46:15021: read: connection reset by peer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.46:15021/healthz/ready": dial tcp 10.132.0.46:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:55 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-54c47cdfc7-lfprg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:06 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-54c47cdfc7-lfprg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:22 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-54c47cdfc7-kgtbh [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:08 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-54c47cdfc7-kgtbh [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:09 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-54c47cdfc7-r2bjb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69 to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:05 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:18 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:18 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:18 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:28 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-kkqwb to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:55 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:55 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:59 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" in 3.706s (3.706s including waiting). Image size: 181251556 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:59 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:59 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:03 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rbgv4 to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:06 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:19 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:25 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.45:15021/healthz/ready": dial tcp 10.132.0.45:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:28 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-s6nj7 to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:08 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:08 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:08 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:08 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:10 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:55 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-kkqwb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:01 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-799f46c59b-kkqwb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:04 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-fqc69 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:05:28 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-799f46c59b-fqc69 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-s6nj7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:14:08 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-799f46c59b-s6nj7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:06 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-rbgv4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:27 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-799f46c59b-rbgv4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:06 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-799f46c59b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:55 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:08 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-54c47cdfc7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:04:59 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:27 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set router-gateway-1-openshift-default-799f46c59b from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:08 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set router-gateway-1-openshift-default-54c47cdfc7 from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:06 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-chk28 to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:18 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.50/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:18 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:18 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:18 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:23 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:17 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-2-openshift-default-54c789bdc6-chk28 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:23 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-2-openshift-default-54c789bdc6-chk28 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-c9txx to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:21 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.51/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:32 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:32 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:32 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:37 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.51:15021/healthz/ready": dial tcp 10.132.0.51:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:20 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-2-openshift-default-684dc9b4b9-c9txx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:17 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-2-openshift-default-54c789bdc6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:17 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:20 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-2-openshift-default-684dc9b4b9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:22 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:23 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set router-gateway-2-openshift-default-54c789bdc6 from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:17 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6 to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.59/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:51 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:51 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:51 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:55 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:55 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:55 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:36:30 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.59:8001/health": dial tcp 10.134.0.59:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:12 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:12 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-664bd66db5-cjjg6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.60/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:54 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:54 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:54 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:36:30 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.60:8000/health": dial tcp 10.134.0.60:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:12 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-prefill-9d68b57d4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8gcvgz to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:51 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.53/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:51 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:51 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:51 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:52 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:52 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:52 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:12 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:12 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-router-scheduler-5c75cd7b8b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-664bd66db5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:33:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-pd-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:34:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:36:50 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [router-with-refs-pd-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:37:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs to ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.52/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:30 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:30 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:30 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:24:16 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.52:8000/health": dial tcp 10.134.0.52:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:25:31 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:25:47 kserve-ci-e2e-test kubelet/ip-10-0-128-215.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.52:8000/health": dial tcp 10.134.0.52:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:25 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-555479964d-4kccs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:38f70cb77701ee00560ecdde553dd84342a4d8f7cb0779aa5b508807d9895bf1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:28 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:28 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:28 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:25:31 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:25:31 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-router-scheduler-7f865578d7-hrblf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-router-scheduler-7f865578d7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:25 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-555479964d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:22:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:24:29 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [router-with-refs-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:25:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:56:58 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:56:58 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:56:58 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:56:58 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:59:08 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:59:23 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.45:8000/health": dial tcp 10.133.0.45:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:56:58 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-configmap-ref-test-kserve-744bf59fbd-chn2n [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:56:58 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87bprxs to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:56:59 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.59/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:56:59 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:56:59 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:56:59 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:59:08 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:59:08 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:56:58 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-configmap-ref-test-kserve-router-scheduler-68f876cc5d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:56:58 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-configmap-ref-test-kserve-744bf59fbd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:56:54 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-configmap-ref-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-configmap-ref-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:56:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:56:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:56:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:56:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:56:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-configmap-ref-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:56:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:56:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:56:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:56:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:57:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:57:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:57:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:58:34 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-configmap-ref-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:59:08 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-configmap-ref-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-28w56 to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:02:24 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:02:24 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:02:24 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:02:24 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:02:23 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-custom-template-test-kserve-5bc7bd6864-28w56 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:02:24 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-custom-template-test-kserve-router-scheduler-567bmx62 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567bmx62 to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:02:24 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.61/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:02:24 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:02:24 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:02:24 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:02:24 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-custom-template-test-kserve-router-scheduler-567469f665 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:02:23 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-custom-template-test-kserve-5bc7bd6864 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:02:20 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-custom-template-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-custom-template-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-custom-template-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:02:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-custom-template-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:02:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-custom-template-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:02:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-custom-template-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:02:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-custom-template-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:02:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-custom-template-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:02:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-custom-template-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:02:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-custom-template-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:02:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:02:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-custom-template-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:02:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-custom-template-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5 to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:00:10 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:00:10 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:00:10 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:00:10 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:01:49 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:00:10 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-5c5cfcf776-9djm5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:00:10 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:00:10 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:00:10 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:00:10 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:01:49 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Warning Unhealthy Readiness probe failed: service unhealthy (responded with "NOT_SERVING") [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:01:49 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:01:49 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:00:10 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:00:10 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9qw9w [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbch5s to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:00:10 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.60/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:00:10 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:00:10 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:00:10 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:01:49 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:01:49 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:00:10 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbc4 from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:00:10 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-ha-replicas-test-kserve-5c5cfcf776 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:00:07 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-ha-replicas-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-ha-replicas-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:00:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:00:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:00:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:00:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:00:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:00:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:00:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:00:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:00:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:01:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:01:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:01:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:01:09 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-ha-replicas-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 20:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s to ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:50:23 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:50:23 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:50:23 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:50:23 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:52:56 kserve-ci-e2e-test kubelet/ip-10-0-139-134.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:50:22 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-inline-config-test-kserve-7c6c58cc96-t4d7s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:50:22 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrbp26 to ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:50:23 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.57/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:50:23 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:50:23 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:50:23 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:52:56 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:52:56 kserve-ci-e2e-test kubelet/ip-10-0-131-203.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:50:22 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-inline-config-test-kserve-router-scheduler-d9ddf6c4d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:50:22 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-inline-config-test-kserve-7c6c58cc96 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:50:21 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-inline-config-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-inline-config-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-inline-config-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:50:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-inline-config-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:50:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-inline-config-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:50:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-inline-config-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:50:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-inline-config-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:50:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-inline-config-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:50:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-inline-config-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:50:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-inline-config-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:50:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:50:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-inline-config-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:51:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-inline-config-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:51:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-inline-config-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:51:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-inline-config-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:52:05 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-inline-config-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-13 19:52:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-inline-config-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:204 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '460e8511-4201-40b3-9364-4a24be200bb5', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Mon, 13 Jul 2026 20:02:58 GMT', 'Content-Length': '236'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" is waiting to start: ContainerCreating","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:204 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '610d5f53-de7a-4763-b7d5-6436f0bdba5e', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Mon, 13 Jul 2026 20:02:58 GMT', 'Content-Length': '238'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" is waiting to start: ContainerCreating","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2 (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 {"level":"info","ts":1783972102.180063,"logger":"setup","caller":"runner/runner.go:196","msg":"GIE build","commit-sha":"181aa8358916e19b8844ccc752b2d6153d4b2ad6","build-ref":"v0.9.0-rc.2"} [e2e-llm-inference-service] Flag --model-server-metrics-scheme has been deprecated, This flag is deprecated. Configure via EndpointPickerConfig data layer plugin parameters instead. [e2e-llm-inference-service] {"level":"info","ts":1783972102.1802306,"logger":"setup","caller":"runner/runner.go:217","msg":"Flags processed","flags":{"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"https","pool-group":"inference.networking.k8s.io","pool-name":"llmisvc-model-pvc-router-manage-2577e794-inference-pool","pool-namespace":"kserve-ci-e2e-test","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783972102.180405,"logger":"setup.trace","caller":"tracing/telemetry.go:123","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.1808734,"caller":"loader/configloader.go:89","msg":"DEPRECATION: apiVersion inference.networking.x-k8s.io/v1alpha1/EndpointPickerConfig is deprecated","replacement":"llm-d.ai/v1alpha1/EndpointPickerConfig"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.1809163,"caller":"loader/configloader.go:121","msg":"Loaded raw configuration","config":"{Plugins: [{Type: single-profile-handler} {Type: queue-scorer} {Type: prefix-cache-scorer} {Type: max-score-picker}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}]}"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.1809282,"logger":"setup","caller":"runner/runner.go:622","msg":"Data layer: ENABLED"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.181213,"logger":"setup","caller":"runner/runner.go:281","msg":"Raw config after phase one","config":{"apiVersion":"inference.networking.x-k8s.io/v1alpha1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"single-profile-handler","parameters":null,"type":"single-profile-handler"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"}],"schedulingProfiles":[{"name":"default","plugins":[{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]}]}} [e2e-llm-inference-service] {"level":"info","ts":1783972102.1965194,"logger":"utilization-detector/utilization-detector","caller":"utilization/detector.go:83","msg":"Creating new UtilizationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms","headroom":0} [e2e-llm-inference-service] {"level":"info","ts":1783972102.1966357,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"vllm","mapping":"Mapping{all specs enabled}"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.196684,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"sglang","mapping":"Mapping{disabled: [lora]}"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.1967223,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"trtllm-serve","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.1967854,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton-tensorrt-llm","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.1968176,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton","mapping":"Mapping{disabled: [kv, lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.1968951,"caller":"loader/configloader.go:154","msg":"Instantiated all plugins and applied system defaults. Effective raw configuration","config":"{Plugins: [{Name: single-profile-handler, Type: single-profile-handler} {Name: queue-scorer, Type: queue-scorer} {Name: prefix-cache-scorer, Type: prefix-cache-scorer} {Name: max-score-picker, Type: max-score-picker} {Name: fcfs-ordering-policy, Type: fcfs-ordering-policy} {Name: global-strict-fairness-policy, Type: global-strict-fairness-policy} {Name: static-usage-limit-policy, Type: static-usage-limit-policy} {Name: openai-parser, Type: openai-parser} {Name: anthropic-parser, Type: anthropic-parser} {Name: vllmhttp-parser, Type: vllmhttp-parser} {Name: utilization-detector, Type: utilization-detector} {Name: metrics-data-source, Type: metrics-data-source} {Name: core-metrics-extractor, Type: core-metrics-extractor}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}], DataLayer: {Sources: [{PluginRef: metrics-data-source, Extractors: [{PluginRef: core-metrics-extractor}]}], Discovery: }, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}]}}"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.196963,"caller":"approximateprefix/plugin.go:88","msg":"Prefix DataProducer initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":2048,"maxPrefixTokensToMatch":131072,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":1783972102.197048,"caller":"approximateprefix/plugin.go:111","msg":"WARNING: configured blockSizeTokens is below the recommended minimum, overriding it.","blockSizeTokens":16,"minimum":64,"issue":"https://github.com/llm-d/llm-d-router/issues/1158"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.1970704,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"approx-prefix-cache-producer/approx-prefix-cache-producer","dataKey":"PrefixCacheMatchInfoDataKey/approx-prefix-cache-producer","consumer":"prefix-cache-scorer"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.1971083,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"token-producer/token-producer","dataKey":"TokenizedPrompt/token-producer","consumer":"approx-prefix-cache-producer"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.1972005,"caller":"runner/runner.go:685","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.197212,"logger":"setup","caller":"runner/runner.go:308","msg":"EPP config after phase two","config":"{SchedulerConfig:{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker}]} SaturationDetector:0xc0002d3400 DataConfig:{Sources:[{Plugin:0xc0004be870 Extractors:[0xc0002d3640]}]} FlowControlConfig: ParserRegistry:0xc0002d3cc0}"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.2132552,"logger":"setup","caller":"runner/runner.go:352","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.2133224,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.213345,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/cmdline"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.2133522,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/profile"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.2133584,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/symbol"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.213365,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.2133718,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.2133787,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/trace"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.2133837,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.2133887,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.2133937,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.2133973,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.2134106,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/plugins/state"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.2134187,"logger":"setup","caller":"runner/runner.go:373","msg":"parsed config","scheduler-config":"{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.2134488,"logger":"setup","caller":"datalayer/runtime.go:99","msg":"Configuring datalayer runtime","numSources":1} [e2e-llm-inference-service] {"level":"info","ts":1783972102.213458,"logger":"setup","caller":"datalayer/runtime.go:118","msg":"Processing source","source":"metrics-data-source","numExtractors":1} [e2e-llm-inference-service] {"level":"info","ts":1783972102.2134733,"logger":"setup","caller":"datalayer/runtime.go:147","msg":"Source configured","source":"metrics-data-source","extractors":["core-metrics-extractor/core-metrics-extractor"]} [e2e-llm-inference-service] {"level":"info","ts":1783972102.2134883,"logger":"setup","caller":"datalayer/runtime.go:206","msg":"Datalayer runtime configured","pollers":1,"notifiers":0,"endpointSources":0} [e2e-llm-inference-service] {"level":"info","ts":1783972102.2134972,"logger":"setup","caller":"runner/runner.go:833","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.2136176,"logger":"setup","caller":"runner/runner.go:721","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":1783972102.213629,"logger":"setup","caller":"runner/runner.go:260","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.213664,"logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.213874,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.214204,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.2142262,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":1783972102.2142184,"logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":1783972102.2152147,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.215566,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.2156997,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":1783972102.2156446,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.215962,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.2193096,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.2195826,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.2198536,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.2229197,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.3164132,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.3164546,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783972102.3164132,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.316478,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783972102.4155161,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.4155505,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783972102.4156866,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-2577e794-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-pvc-router-manage-2577e794-inference-pool","reconcileID":"6cda5bc1-b641-4099-97f1-8cf88003132c"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.4165652,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":1783972102.4166038,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783972137.0632138,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-2577e794-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-pvc-router-manage-2577e794-inference-pool","reconcileID":"15e83cbc-2dcb-46d9-bff7-abf5aab68fef"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 4e7a429d-fd3e-420e-8387-2965683dc85f [e2e-llm-inference-service] resourceVersion: '58515' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-13T19:48:54Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:54Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.133.0.42 [e2e-llm-inference-service] nodeName: ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2 [e2e-llm-inference-service] uid: 2ce120d6-0e30-45b7-90fc-f7f76654786e [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 6c927af0-0231-4267-ac61-a75b8ae1cc0f [e2e-llm-inference-service] resourceVersion: '58029' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 2a700b56-6140-4f86-817f-877ba9bd95ce [e2e-llm-inference-service] resourceVersion: '58056' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-leader [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] apps.kubernetes.io/pod-index: '0' [e2e-llm-inference-service] controller-revision-hash: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-7b97d8cc8b [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: dc52feed9d7e0b6abd87bf4d45c9409765943cd8 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 686d468674 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '0' [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] statefulset.kubernetes.io/pod-name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.56/23"],"mac_address":"0a:58:0a:84:00:38","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.56/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/size: '2' [e2e-llm-inference-service] openshift.io/scc: openshift-ai-llminferenceservice-scc [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: serviceaccount [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: StatefulSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] uid: d52437e4-3c15-4c37-89b8-6db6b4f4c498 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-131-203 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/size: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:apps.kubernetes.io/pod-index: {} [e2e-llm-inference-service] f:controller-revision-hash: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/template-revision-hash: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/worker-index: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:statefulset.kubernetes.io/pod-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d52437e4-3c15-4c37-89b8-6db6b4f4c498"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:add: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:hostname: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:subdomain: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-7zc7x [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: LWS_LEADER_ADDRESS [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0.llmisvc-model-pvc-router-manage-2577e794-kserve-mn.kserve-ci-e2e-test [e2e-llm-inference-service] - name: LWS_GROUP_SIZE [e2e-llm-inference-service] value: '2' [e2e-llm-inference-service] - name: LWS_WORKER_INDEX [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-7zc7x [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] nodeName: ip-10-0-131-203.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-gv2sm [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-8xddh [e2e-llm-inference-service] hostname: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] subdomain: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] hostIP: 10.0.131.203 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.131.203 [e2e-llm-inference-service] startTime: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: ContainerCreating [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-7zc7x [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: a4b3b3ac-0484-42d4-a875-f74db7379ea1 [e2e-llm-inference-service] resourceVersion: '58065' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] apps.kubernetes.io/pod-index: '1' [e2e-llm-inference-service] controller-revision-hash: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-579bcdf848 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: dc52feed9d7e0b6abd87bf4d45c9409765943cd8 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 686d468674 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '1' [e2e-llm-inference-service] statefulset.kubernetes.io/pod-name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.63/23"],"mac_address":"0a:58:0a:86:00:3f","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.63/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/leader-name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/size: '2' [e2e-llm-inference-service] openshift.io/scc: openshift-ai-llminferenceservice-scc [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: serviceaccount [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: StatefulSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] uid: 52ce2340-1390-4cd9-be8d-ba0d52ededfd [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-128-215 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/leader-name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/size: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:apps.kubernetes.io/pod-index: {} [e2e-llm-inference-service] f:controller-revision-hash: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/group-index: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/group-key: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/template-revision-hash: {} [e2e-llm-inference-service] f:statefulset.kubernetes.io/pod-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"52ce2340-1390-4cd9-be8d-ba0d52ededfd"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:add: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:hostname: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:subdomain: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-986v7 [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: LWS_LEADER_ADDRESS [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0.llmisvc-model-pvc-router-manage-2577e794-kserve-mn.kserve-ci-e2e-test [e2e-llm-inference-service] - name: LWS_GROUP_SIZE [e2e-llm-inference-service] value: '2' [e2e-llm-inference-service] - name: LWS_WORKER_INDEX [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-986v7 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] nodeName: ip-10-0-128-215.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-gv2sm [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-8xddh [e2e-llm-inference-service] hostname: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] subdomain: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] hostIP: 10.0.128.215 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.128.215 [e2e-llm-inference-service] startTime: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: ContainerCreating [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-986v7 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2 [e2e-llm-inference-service] generateName: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-7dbcb75dbc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 2ce120d6-0e30-45b7-90fc-f7f76654786e [e2e-llm-inference-service] resourceVersion: '58513' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7dbcb75dbc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.42/23"],"mac_address":"0a:58:0a:85:00:2a","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.42/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.42\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:2a\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-7dbcb75dbc [e2e-llm-inference-service] uid: 0355a2c5-07c9-4036-9c85-cda550978776 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-139-134 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"0355a2c5-07c9-4036-9c85-cda550978776"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:22Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:54Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.42"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-hrxgx [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: prefix-cache-scorer\n\ [e2e-llm-inference-service] - type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-hrxgx [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-gv2sm [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-hnhlw [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:48:23Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:48:54Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:48:54Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] hostIP: 10.0.139.134 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.139.134 [e2e-llm-inference-service] podIP: 10.133.0.42 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.42 [e2e-llm-inference-service] startTime: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-13T19:48:22Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-router-endpoint-picker@sha256:06b6c75d77afd0e07053402752a9736c2dfbc12a306d0d37d963aac4c1d4e6a6 [e2e-llm-inference-service] containerID: cri-o://9ec5f95b130b147427c28f79970902aaa9cdc5d762b85f89422eeae59552326b [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-hrxgx [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 8b0be98a-0382-4743-9825-125467f92a82 [e2e-llm-inference-service] resourceVersion: '58038' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-hnhlw [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: d209db60-7b81-4f60-9cb9-be0ce92a99c3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-hnhlw"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d209db60-7b81-4f60-9cb9-be0ce92a99c3"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-gv2sm"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-gv2sm [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-hnhlw [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-gv2sm [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-hnhlw [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 82e7143a-763d-4965-998b-0c26ba37588e [e2e-llm-inference-service] resourceVersion: '58016' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-8xddh [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: d209db60-7b81-4f60-9cb9-be0ce92a99c3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-8xddh"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d209db60-7b81-4f60-9cb9-be0ce92a99c3"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-gv2sm"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-gv2sm [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-8xddh [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-gv2sm [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-8xddh [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 9a7c7ad4-a5d4-461b-a736-85dd2d32dd9f [e2e-llm-inference-service] resourceVersion: '58077' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: d209db60-7b81-4f60-9cb9-be0ce92a99c3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d209db60-7b81-4f60-9cb9-be0ce92a99c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.16.245 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.16.245 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: d10073d4-36c9-489c-bd89-d72959302567 [e2e-llm-inference-service] resourceVersion: '58025' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: d209db60-7b81-4f60-9cb9-be0ce92a99c3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d209db60-7b81-4f60-9cb9-be0ce92a99c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.237.189 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.237.189 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-579bcdf848 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 68584626-bb45-4316-8b4f-016cc4cc5a7d [e2e-llm-inference-service] resourceVersion: '58051' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] controller.kubernetes.io/hash: 579bcdf848 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: dc52feed9d7e0b6abd87bf4d45c9409765943cd8 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 686d468674 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: StatefulSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] uid: 52ce2340-1390-4cd9-be8d-ba0d52ededfd [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:data: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:controller.kubernetes.io/hash: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/group-index: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/group-key: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/template-revision-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"52ce2340-1390-4cd9-be8d-ba0d52ededfd"}: {} [e2e-llm-inference-service] f:revision: {} [e2e-llm-inference-service] data: [e2e-llm-inference-service] spec: [e2e-llm-inference-service] template: [e2e-llm-inference-service] $patch: replace [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/leader-name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/size: '2' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: dc52feed9d7e0b6abd87bf4d45c9409765943cd8 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 686d468674 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] name: main [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /home [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - mountPath: /tmp [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - mountPath: /dev/shm [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - mountPath: /models [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - mountPath: /var/run/kserve/tls [e2e-llm-inference-service] name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] revision: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ControllerRevision [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-7b97d8cc8b [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 566ab45c-a1ea-4ad5-94ee-d89e86137de3 [e2e-llm-inference-service] resourceVersion: '58028' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-leader [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] controller.kubernetes.io/hash: 7b97d8cc8b [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 686d468674 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '0' [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/replicas: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: StatefulSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] uid: d52437e4-3c15-4c37-89b8-6db6b4f4c498 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:data: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/replicas: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:controller.kubernetes.io/hash: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/template-revision-hash: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/worker-index: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d52437e4-3c15-4c37-89b8-6db6b4f4c498"}: {} [e2e-llm-inference-service] f:revision: {} [e2e-llm-inference-service] data: [e2e-llm-inference-service] spec: [e2e-llm-inference-service] template: [e2e-llm-inference-service] $patch: replace [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/size: '2' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-leader [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 686d468674 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '0' [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] name: main [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /home [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - mountPath: /tmp [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - mountPath: /dev/shm [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - mountPath: /models [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - mountPath: /var/run/kserve/tls [e2e-llm-inference-service] name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] revision: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ControllerRevision [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 0ca83935-b08d-45d3-aef8-0e1baa9a1de1 [e2e-llm-inference-service] resourceVersion: '58517' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: d209db60-7b81-4f60-9cb9-be0ce92a99c3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d209db60-7b81-4f60-9cb9-be0ce92a99c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:54Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n-\ [e2e-llm-inference-service] \ name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-13T19:48:54Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:48:54Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-13T19:48:54Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-7dbcb75dbc" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-7dbcb75dbc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 0355a2c5-07c9-4036-9c85-cda550978776 [e2e-llm-inference-service] resourceVersion: '58516' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7dbcb75dbc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler [e2e-llm-inference-service] uid: 0ca83935-b08d-45d3-aef8-0e1baa9a1de1 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"0ca83935-b08d-45d3-aef8-0e1baa9a1de1"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:54Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7dbcb75dbc [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7dbcb75dbc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n-\ [e2e-llm-inference-service] \ name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: b16cee8d-1b6d-4ce4-819c-36bfb39fe050 [e2e-llm-inference-service] resourceVersion: '58068' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: d209db60-7b81-4f60-9cb9-be0ce92a99c3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d209db60-7b81-4f60-9cb9-be0ce92a99c3"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-scc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 4cef6f76-d27f-4380-9ab2-45429d053c2f [e2e-llm-inference-service] resourceVersion: '58012' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: d209db60-7b81-4f60-9cb9-be0ce92a99c3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d209db60-7b81-4f60-9cb9-be0ce92a99c3"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-prefill [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: ClusterRole [e2e-llm-inference-service] name: openshift-ai-llminferenceservice-scc [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 86db5b7b-bb40-497f-91cd-c4bc5e91dcbd [e2e-llm-inference-service] resourceVersion: '58064' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: d209db60-7b81-4f60-9cb9-be0ce92a99c3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d209db60-7b81-4f60-9cb9-be0ce92a99c3"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service-bghch [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-2577e794-epp-service- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: d3bd52e8-d05e-4ea5-8013-bedda03297c5 [e2e-llm-inference-service] resourceVersion: '58514' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-13T19:48:54Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] uid: 9a7c7ad4-a5d4-461b-a736-85dd2d32dd9f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:54Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"9a7c7ad4-a5d4-461b-a736-85dd2d32dd9f"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.133.0.42 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2 [e2e-llm-inference-service] uid: 2ce120d6-0e30-45b7-90fc-f7f76654786e [e2e-llm-inference-service] nodeName: ip-10-0-139-134.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-srvbfq [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 7bb8f2bb-078a-46dc-9544-4d1f6cecb91d [e2e-llm-inference-service] resourceVersion: '58030' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] uid: d10073d4-36c9-489c-bd89-d72959302567 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d10073d4-36c9-489c-bd89-d72959302567"}: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: null [e2e-llm-inference-service] ports: null [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: b16cee8d-1b6d-4ce4-819c-36bfb39fe050 [e2e-llm-inference-service] resourceVersion: '58068' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: d209db60-7b81-4f60-9cb9-be0ce92a99c3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d209db60-7b81-4f60-9cb9-be0ce92a99c3"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-scc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 4cef6f76-d27f-4380-9ab2-45429d053c2f [e2e-llm-inference-service] resourceVersion: '58012' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: d209db60-7b81-4f60-9cb9-be0ce92a99c3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d209db60-7b81-4f60-9cb9-be0ce92a99c3"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:llmisvc-model-pvc-router-manage-2577e794-kserve-mn-prefill [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-prefill [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] name: openshift-ai-llminferenceservice-scc [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 86db5b7b-bb40-497f-91cd-c4bc5e91dcbd [e2e-llm-inference-service] resourceVersion: '58064' [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: d209db60-7b81-4f60-9cb9-be0ce92a99c3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d209db60-7b81-4f60-9cb9-be0ce92a99c3"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d209db60-7b81-4f60-9cb9-be0ce92a99c3"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:48:57Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-13T19:48:57Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-13T19:48:58Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: d209db60-7b81-4f60-9cb9-be0ce92a99c3 [e2e-llm-inference-service] resourceVersion: '58580' [e2e-llm-inference-service] uid: c497556f-f221-4867-9ff5-58bd3264751b [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:48:57Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:48:22Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d209db60-7b81-4f60-9cb9-be0ce92a99c3"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:48:57Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-13T19:48:57Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-13T19:48:58Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: d209db60-7b81-4f60-9cb9-be0ce92a99c3 [e2e-llm-inference-service] resourceVersion: '58580' [e2e-llm-inference-service] uid: c497556f-f221-4867-9ff5-58bd3264751b [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:48:57Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:48:22Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d209db60-7b81-4f60-9cb9-be0ce92a99c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-13T19:48:57Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: d209db60-7b81-4f60-9cb9-be0ce92a99c3 [e2e-llm-inference-service] resourceVersion: '58556' [e2e-llm-inference-service] uid: 4a4a7342-6e1b-4185-85d5-c7ddcd305619 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:48:57Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:48:57Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:24Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d209db60-7b81-4f60-9cb9-be0ce92a99c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:48:24Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-13T19:48:26Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: d209db60-7b81-4f60-9cb9-be0ce92a99c3 [e2e-llm-inference-service] resourceVersion: '58234' [e2e-llm-inference-service] uid: b75ebbb9-9d86-4951-8ca8-f3d8d772e255 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:48:25Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:48:26Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: leaderworkerset.x-k8s.io/v1 [e2e-llm-inference-service] kind: LeaderWorkerSet [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: leaderworkerset.x-k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:hpaPodSelector: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] manager: lws [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] - apiVersion: leaderworkerset.x-k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d209db60-7b81-4f60-9cb9-be0ce92a99c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderWorkerTemplate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderTemplate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:add: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:size: {} [e2e-llm-inference-service] f:workerTemplate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:add: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:rolloutStrategy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupPolicy: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: d209db60-7b81-4f60-9cb9-be0ce92a99c3 [e2e-llm-inference-service] resourceVersion: '58055' [e2e-llm-inference-service] uid: d9e8b688-a8ca-4dc1-b42c-76679368aa4d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] leaderWorkerTemplate: [e2e-llm-inference-service] leaderTemplate: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-leader [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] name: main [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /home [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - mountPath: /tmp [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - mountPath: /dev/shm [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - mountPath: /models [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - mountPath: /var/run/kserve/tls [e2e-llm-inference-service] name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] restartPolicy: RecreateGroupOnPodRestart [e2e-llm-inference-service] size: 2 [e2e-llm-inference-service] workerTemplate: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] spec: [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] name: main [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /home [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - mountPath: /tmp [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - mountPath: /dev/shm [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - mountPath: /models [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - mountPath: /var/run/kserve/tls [e2e-llm-inference-service] name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] networkConfig: [e2e-llm-inference-service] subdomainPolicy: Shared [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] rolloutStrategy: [e2e-llm-inference-service] rollingUpdateConfiguration: [e2e-llm-inference-service] maxSurge: 0 [e2e-llm-inference-service] maxUnavailable: 1 [e2e-llm-inference-service] partition: 0 [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] startupPolicy: LeaderCreated [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] message: Replicas are progressing [e2e-llm-inference-service] reason: GroupsProgressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Progressing [e2e-llm-inference-service] hpaPodSelector: leaderworkerset.sigs.k8s.io/name=llmisvc-model-pvc-router-manage-2577e794-kserve-mn,leaderworkerset.sigs.k8s.io/worker-index=0 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:22Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d209db60-7b81-4f60-9cb9-be0ce92a99c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:48:22Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: d209db60-7b81-4f60-9cb9-be0ce92a99c3 [e2e-llm-inference-service] resourceVersion: '58114' [e2e-llm-inference-service] uid: a3f22566-3a06-4eac-a508-4fdabd2fad84 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:57Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d209db60-7b81-4f60-9cb9-be0ce92a99c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:48:57Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: d209db60-7b81-4f60-9cb9-be0ce92a99c3 [e2e-llm-inference-service] resourceVersion: '58560' [e2e-llm-inference-service] uid: 6ce4a8d7-56bf-479d-a83e-62c2e34342a1 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-inference--ip-950f33a3.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:22Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d209db60-7b81-4f60-9cb9-be0ce92a99c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:48:22Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: d209db60-7b81-4f60-9cb9-be0ce92a99c3 [e2e-llm-inference-service] resourceVersion: '58116' [e2e-llm-inference-service] uid: dbe5a574-ecdd-48e7-830a-3cdcceccd18a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:22Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d209db60-7b81-4f60-9cb9-be0ce92a99c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:48:22Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: d209db60-7b81-4f60-9cb9-be0ce92a99c3 [e2e-llm-inference-service] resourceVersion: '58114' [e2e-llm-inference-service] uid: a3f22566-3a06-4eac-a508-4fdabd2fad84 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:57Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d209db60-7b81-4f60-9cb9-be0ce92a99c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:48:57Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: d209db60-7b81-4f60-9cb9-be0ce92a99c3 [e2e-llm-inference-service] resourceVersion: '58560' [e2e-llm-inference-service] uid: 6ce4a8d7-56bf-479d-a83e-62c2e34342a1 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-inference--ip-950f33a3.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:22Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d209db60-7b81-4f60-9cb9-be0ce92a99c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:48:22Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: d209db60-7b81-4f60-9cb9-be0ce92a99c3 [e2e-llm-inference-service] resourceVersion: '58116' [e2e-llm-inference-service] uid: dbe5a574-ecdd-48e7-830a-3cdcceccd18a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:22Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d209db60-7b81-4f60-9cb9-be0ce92a99c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:48:22Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: d209db60-7b81-4f60-9cb9-be0ce92a99c3 [e2e-llm-inference-service] resourceVersion: '58114' [e2e-llm-inference-service] uid: a3f22566-3a06-4eac-a508-4fdabd2fad84 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:57Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d209db60-7b81-4f60-9cb9-be0ce92a99c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:48:57Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: d209db60-7b81-4f60-9cb9-be0ce92a99c3 [e2e-llm-inference-service] resourceVersion: '58560' [e2e-llm-inference-service] uid: 6ce4a8d7-56bf-479d-a83e-62c2e34342a1 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-inference--ip-950f33a3.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:22Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d209db60-7b81-4f60-9cb9-be0ce92a99c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:48:22Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: d209db60-7b81-4f60-9cb9-be0ce92a99c3 [e2e-llm-inference-service] resourceVersion: '58116' [e2e-llm-inference-service] uid: dbe5a574-ecdd-48e7-830a-3cdcceccd18a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d209db60-7b81-4f60-9cb9-be0ce92a99c3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-13T19:48:21Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: d209db60-7b81-4f60-9cb9-be0ce92a99c3 [e2e-llm-inference-service] resourceVersion: '58088' [e2e-llm-inference-service] uid: b5a4ce7d-0be0-4353-8b83-1dea3e219290 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-07-13T20:02:59Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7dbcb75dbc [e2e-llm-inference-service] timestamp: '2026-07-13T20:02:45Z' [e2e-llm-inference-service] window: 17.089s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 725847n [e2e-llm-inference-service] memory: 20720Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-07-13T20:02:59.215888] end - ❌ 902.454s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-13T19:48:59Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-13T19:48:59Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-13T19:48:24Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-13T19:48:59Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-13T19:48:59Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-13T19:48:59Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:48:24Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-13T19:48:24Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] =============================== warnings summary =============================== [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] /workspace/source/python/kserve/.venv/lib64/python3.11/site-packages/pytest_asyncio/plugin.py:761: DeprecationWarning: The event_loop fixture provided by pytest-asyncio has been redefined in [e2e-llm-inference-service] /workspace/source/test/e2e/conftest.py:43 [e2e-llm-inference-service] Replacing the event_loop fixture with a custom implementation is deprecated [e2e-llm-inference-service] and will lead to errors in the future. [e2e-llm-inference-service] If you want to request an asyncio event loop with a scope other than function [e2e-llm-inference-service] scope, use the "scope" argument to the asyncio mark when marking the tests. [e2e-llm-inference-service] If you want to return different types of event loops, use the event_loop_policy [e2e-llm-inference-service] fixture. [e2e-llm-inference-service] [e2e-llm-inference-service] warnings.warn( [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py:40: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_tls.py:93: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html [e2e-llm-inference-service] ---------- generated xml file: /workspace/artifacts-dir/junit_e2e.xml ---------- [e2e-llm-inference-service] --------------------------------- JSON report ---------------------------------- [e2e-llm-inference-service] report saved to: /workspace/artifacts-dir/e2e_results.json [e2e-llm-inference-service] =========================== short test summary info ============================ [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] ====== 3 failed, 39 passed, 3 skipped, 29 warnings in 4681.56s (1:18:01) ======= [must-gather] [must-gather ] OUT 2026-07-13T20:22:55.448774349Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 79ef62dc-8bf7-4a50-8cc9-d8dfbff73685 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.24" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-13T20:22:55.498135261Z namespace/openshift-must-gather-stwlp created [must-gather] [must-gather ] OUT 2026-07-13T20:22:55.50521568Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-wrjnw created [must-gather] [must-gather ] OUT 2026-07-13T20:22:55.533438866Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:02.500847424Z [disk usage checker] Started [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:02.504153135Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:02.773163844Z Error from server (NotFound): namespaces "redhat-ods-operator" not found [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:02.945448368Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:02.945481002Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:02.945481002Z namespaces "redhat-ods-operator" not found [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:02.947943737Z Error getting logs from redhat-ods-operator [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:03.122166346Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:03.122197242Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:03.122197242Z namespaces "redhat-ods-monitoring" not found [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:03.124428348Z Error getting logs from redhat-ods-monitoring [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:03.295683223Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:03.295717405Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:03.295717405Z namespaces "redhat-ods-applications" not found [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:03.298786042Z Error getting logs from redhat-ods-applications [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:03.466715049Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:03.466747244Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:03.466747244Z namespaces "rhods-notebooks" not found [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:03.469053150Z Error getting logs from rhods-notebooks [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:03.636491731Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:03.636523735Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:03.636523735Z namespaces "rhoai-model-registries" not found [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:03.638771800Z Error getting logs from rhoai-model-registries [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:03.806408538Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:03.806508779Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:03.806508779Z namespaces "istio-system" not found [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:03.808623795Z Error getting logs from istio-system [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:03.973169957Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:03.973201091Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:03.973201091Z namespaces "knative-serving" not found [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:03.975359629Z Error getting logs from knative-serving [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:04.138766713Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:04.138797641Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:04.138797641Z namespaces "redhat-ods-applications-auth-provider" not found [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:04.141137412Z Error getting logs from redhat-ods-applications-auth-provider [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:04.431106590Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:04.724731458Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:04.862833966Z error: the server doesn't have a resource type "auths" [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:04.954032900Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:04.956208071Z Error collecting info from [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:05.096836257Z error: the server doesn't have a resource type "monitorings" [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:05.185486733Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:05.187501111Z Error collecting info from [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:05.324325201Z error: the server doesn't have a resource type "featuretrackers" [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:05.414969783Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:05.417044346Z Error collecting info from [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:05.553721619Z error: the server doesn't have a resource type "codeflares" [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:05.645537607Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:05.647689891Z Error collecting info from [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:05.786152969Z error: the server doesn't have a resource type "dashboards" [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:05.873762551Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:05.875900677Z Error collecting info from [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:06.013854441Z error: the server doesn't have a resource type "datasciencepipelines" [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:06.101240993Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:06.103231938Z Error collecting info from [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:06.242121967Z error: the server doesn't have a resource type "feastoperators" [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:06.329820124Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:06.331996549Z Error collecting info from [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:06.469463479Z error: the server doesn't have a resource type "kserves" [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:06.555950974Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:06.558028413Z Error collecting info from [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:06.691847429Z error: the server doesn't have a resource type "kueues" [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:06.779724009Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:06.782022794Z Error collecting info from [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:06.925716324Z error: the server doesn't have a resource type "modelcontrollers" [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:07.015133043Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:07.017379743Z Error collecting info from [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:07.150108773Z error: the server doesn't have a resource type "modelmeshservings" [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:07.238051249Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:07.240111567Z Error collecting info from [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:07.377394665Z error: the server doesn't have a resource type "modelregistries" [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:07.462279391Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:07.464202868Z Error collecting info from [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:07.508814590Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:07.601140189Z error: the server doesn't have a resource type "rays" [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:07.685331495Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:07.687475463Z Error collecting info from [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:07.825699343Z error: the server doesn't have a resource type "trainingoperators" [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:07.911904429Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:07.914153155Z Error collecting info from [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:08.048812177Z error: the server doesn't have a resource type "trustyais" [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:08.135567310Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:08.137972938Z Error collecting info from [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:08.273277739Z error: the server doesn't have a resource type "workbenches" [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:08.360935711Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:08.363162096Z Error collecting info from [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:08.499419960Z error: the server doesn't have a resource type "hardwareprofiles" [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:08.586527111Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:08.588569015Z Error collecting info from [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:08.720381259Z error: the server doesn't have a resource type "llamastackoperators" [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:08.807759668Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:08.809913812Z Error collecting info from [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:09.543316064Z error: the server doesn't have a resource type "predictors" [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:09.670251714Z error: the server doesn't have a resource type "localmodelnodegroups" [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:09.804567337Z error: the server doesn't have a resource type "smcp" [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:09.941470556Z error: the server doesn't have a resource type "smm" [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:10.074641674Z error: the server doesn't have a resource type "smmr" [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:10.687249809Z error: the server doesn't have a resource type "knativeservings" [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:10.820476557Z error: the server doesn't have a resource type "configurations" [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:10.959434002Z error: the server doesn't have a resource type "routes" [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:11.095424956Z error: the server doesn't have a resource type "services" [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:11.231140493Z error: the server doesn't have a resource type "revisions" [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:11.811923917Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:12.513500269Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:12.553401485Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:13.400601271Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:13.794011908Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:13.794064980Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:13.794064980Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:13.794064980Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:13.794064980Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:13.794064980Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:13.794064980Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:13.794064980Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:13.794064980Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:13.794064980Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:13.794064980Z [container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:13.794064980Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:13.794064980Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:13.794064980Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:13.794064980Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" not found]] [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:13.798598740Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:13.966697371Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:14.462517579Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:15.253645150Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:15.639155631Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:15.639199343Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:15.639199343Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:15.639199343Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:15.639199343Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:15.639199343Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:15.639199343Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:15.639199343Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:15.639199343Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:15.639199343Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:15.639199343Z [container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:15.639199343Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:15.639199343Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:15.639199343Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:15.639199343Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" not found]] [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:15.644194838Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:15.811428752Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:16.403761801Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:17.271723504Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:17.518178864Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:17.695469934Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:17.695520995Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:17.695520995Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:17.695520995Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:17.695520995Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:17.695520995Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:17.695520995Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:17.695520995Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:17.695520995Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:17.695520995Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:17.695520995Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" not found, container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:17.695520995Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:17.695520995Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:17.695520995Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:17.695520995Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" not found]] [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:17.700141527Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:17.879438762Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:18.270034285Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:19.083471501Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:19.484318462Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:19.484381633Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:19.484381633Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:19.484381633Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:19.484381633Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:19.484381633Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:19.484381633Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:19.484381633Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:19.484381633Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:19.484381633Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:19.484381633Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" not found, container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:19.484381633Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:19.484381633Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:19.484381633Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:19.484381633Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" not found]] [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:19.489428183Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:19.668988230Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:20.085556034Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:20.913510112Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:21.325713340Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:21.325762700Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:21.325762700Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:21.325762700Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:21.325762700Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:21.325762700Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:21.325762700Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:21.325762700Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:21.325762700Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:21.325762700Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:21.325762700Z [container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:21.325762700Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:21.325762700Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:21.325762700Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:21.325762700Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" not found]] [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:21.331517100Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:21.510530034Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:21.941494659Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:22.524746874Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:22.748829956Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:23.109887794Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:23.109934046Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:23.109934046Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:23.109934046Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:23.109934046Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:23.109934046Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:23.109934046Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:23.109934046Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:23.109934046Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:23.109934046Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:23.109934046Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" not found, container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:23.109934046Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:23.109934046Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:23.109934046Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:23.109934046Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" not found]] [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:23.114262146Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:23.276788035Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:23.788977601Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:24.618526462Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:24.998961363Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:24.999004032Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:24.999004032Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:24.999004032Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:24.999004032Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:24.999004032Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:24.999004032Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:24.999004032Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:24.999004032Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:24.999004032Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:24.999004032Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" not found, container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:24.999004032Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:24.999004032Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:24.999004032Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:24.999004032Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" is waiting to start: ContainerCreating]] [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:25.004517440Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:25.173446682Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:25.544742220Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:26.303846290Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:26.738283085Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:26.738363762Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:26.738363762Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:26.738363762Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:26.738363762Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:26.738363762Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:26.738363762Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:26.738363762Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:26.738363762Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:26.738363762Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:26.738363762Z [container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:26.738363762Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:26.738363762Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:26.738363762Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:26.738363762Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" not found]] [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:26.743861326Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:26.911063117Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:27.300863257Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:27.529718770Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:28.054597654Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:28.407739354Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:28.407783499Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:28.407783499Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:28.407783499Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:28.407783499Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:28.407783499Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:28.407783499Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:28.407783499Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:28.407783499Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:28.407783499Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:28.407783499Z [container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:28.407783499Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:28.407783499Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:28.407783499Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:28.407783499Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" not found]] [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:28.412651212Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:28.579418330Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:29.013003249Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:29.787910839Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:30.175484735Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:30.175532899Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:30.175532899Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:30.175532899Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:30.175532899Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:30.175532899Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:30.175532899Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:30.175532899Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:30.175532899Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:30.175532899Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:30.175532899Z [container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:30.175532899Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:30.175532899Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:30.175532899Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:30.175532899Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" not found]] [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:30.180099201Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:30.347965554Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:30.765001614Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:31.532765314Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:31.919615105Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:31.919660326Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:31.919660326Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:31.919660326Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:31.919660326Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:31.919660326Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:31.919660326Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:31.919660326Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:31.919660326Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:31.919660326Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:31.919660326Z [container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:31.919660326Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:31.919660326Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:31.919660326Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:31.919660326Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" is waiting to start: ContainerCreating]] [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:31.925022529Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:32.091424170Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:32.506035837Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:32.534545808Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:33.279328188Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:33.654428678Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:33.654475141Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:33.654475141Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:33.654475141Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:33.654475141Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:33.654475141Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:33.654475141Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:33.654475141Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:33.654475141Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:33.654475141Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:33.654475141Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" not found, container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:33.654475141Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:33.654475141Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:33.654475141Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:33.654475141Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" not found]] [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:33.660282458Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:33.826748670Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:34.264560644Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:35.021240999Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:35.379428559Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:35.379478437Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:35.379478437Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:35.379478437Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:35.379478437Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:35.379478437Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:35.379478437Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:35.379478437Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:35.379478437Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:35.379478437Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:35.379478437Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" not found, container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:35.379478437Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:35.379478437Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:35.379478437Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:35.379478437Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" not found]] [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:35.383535849Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:35.549643835Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:35.924458245Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:36.696138853Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:37.074996247Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:37.075042550Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:37.075042550Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:37.075042550Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:37.075042550Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:37.075042550Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:37.075042550Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:37.075042550Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:37.075042550Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:37.075042550Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:37.075042550Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" not found, container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:37.075042550Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:37.075042550Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:37.075042550Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:37.075042550Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" not found]] [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:37.080497430Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:37.247955474Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:37.539211888Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:37.731824128Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:38.487132269Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:38.873594017Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:38.873638829Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:38.873638829Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:38.873638829Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:38.873638829Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:38.873638829Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:38.873638829Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:38.873638829Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:38.873638829Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:38.873638829Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:38.873638829Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" not found, container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:38.873638829Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:38.873638829Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:38.873638829Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:38.873638829Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" not found]] [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:38.877926462Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:39.048825679Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:39.453023244Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:40.221927922Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:40.612919982Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:40.612968673Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:40.612968673Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:40.612968673Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:40.612968673Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:40.612968673Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:40.612968673Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:40.612968673Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:40.612968673Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:40.612968673Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:40.612968673Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" not found, container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:40.612968673Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:40.612968673Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt: [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:40.612968673Z [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:40.612968673Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" not found]] [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:40.617878130Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:40.789644470Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:41.692880779Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:41.865164818Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:42.250528810Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:42.420696980Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:42.544071327Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:43.358954573Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:43.561395900Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:43.946797898Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:44.121005002Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:45.025480092Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:45.180751584Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:45.572170379Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:45.741338521Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:46.625040027Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:46.776518571Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:47.159103548Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:47.336313554Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:47.548925640Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:48.029596107Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:48.245626824Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:48.659701032Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:48.829876514Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:49.495355968Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:49.723940823Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:50.163256241Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:50.329094278Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:50.995388283Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:51.188573301Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:51.580471911Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:51.746710169Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:52.417860005Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:52.553636521Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:52.613090087Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:53.030165493Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:53.195942295Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:53.867264344Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:54.070891119Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:54.467512929Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:54.638070727Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:55.284919057Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:55.476559708Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:55.878711591Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:56.049033960Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:56.769520196Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:56.954046376Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:57.359890897Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:57.529793227Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:57.558840662Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:58.206764799Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:58.455302674Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:58.854916307Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:59.023849310Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:59.657327191Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:23:59.843527965Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:00.250165030Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:00.418465723Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:01.112870086Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:01.313350559Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:01.719608331Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:01.890398780Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:02.548478701Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:02.563467355Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:02.772872823Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:03.183016330Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:03.351857137Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:04.100531469Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:04.333734394Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:04.738905451Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:04.904761192Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:05.667357165Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:05.886533513Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:06.290527631Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:06.465631074Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:07.152256349Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:07.348522003Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:07.568087206Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:07.746392986Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:07.913587355Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:08.569922535Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:08.796326150Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:09.195674955Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:09.374430213Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:10.107709640Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:10.302872244Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:10.722699744Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:10.902977324Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:11.590655123Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:11.786106583Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:12.193113898Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:12.372578831Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:12.572931750Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:13.100267042Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:13.308223909Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:13.728320592Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:13.909928291Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:14.721067376Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:14.922533092Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:15.320086239Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:15.490934454Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:16.159592635Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:16.357594596Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:16.761139568Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:16.934794473Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:17.577896452Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:17.761120960Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:17.973548190Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:18.383428097Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:18.552268822Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:19.228876047Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:19.430825451Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:19.821660970Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:19.985617326Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:20.695584292Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:20.906247625Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:21.324655812Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:21.504590294Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:22.357928046Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:22.580853538Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:22.582819800Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:23.002348697Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:23.187489233Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:23.928223429Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:24.150565803Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:24.560691090Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:24.731280661Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:25.567970015Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:25.778795946Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:26.180845112Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:26.352024713Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:27.041422235Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:27.228848932Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:27.588067459Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:27.632855498Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:27.815530543Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:28.501855278Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:28.704235206Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:29.115159178Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:29.301612832Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:30.067230419Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:30.275892210Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:30.694152559Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:30.867615369Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:31.663495293Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:31.905970015Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:32.317248739Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:32.507706702Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:32.597394184Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:33.400275771Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:33.622684164Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:34.051283147Z Wrote inspect data to must-gather. [must-gather] [must-gather-wkpst] POD 2026-07-13T20:24:34.075056081Z Caches written to disk [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.083844704Z waiting for gather to complete [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.088591375Z downloading gather output [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.419206831Z receiving incremental file list [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.445284278Z ./ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.445453533Z aggregated-discovery-api.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.445837335Z aggregated-discovery-apis.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.446888026Z event-filter.html [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.449508024Z timestamp [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.449660359Z version [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.449684159Z cluster-scoped-resources/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.44969667Z cluster-scoped-resources/datasciencecluster.opendatahub.io/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.44970305Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.449751672Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/test-dsc.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.449819843Z cluster-scoped-resources/dscinitialization.opendatahub.io/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.449831884Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.449845554Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/test-dsci.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.449921157Z namespaces/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.449930427Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.449959798Z namespaces/kserve-ci-e2e-test/kserve-ci-e2e-test.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.45004109Z namespaces/kserve-ci-e2e-test/apps.openshift.io/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.450063791Z namespaces/kserve-ci-e2e-test/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.450127133Z namespaces/kserve-ci-e2e-test/apps/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.450159504Z namespaces/kserve-ci-e2e-test/apps/daemonsets.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.450271757Z namespaces/kserve-ci-e2e-test/apps/deployments.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.450796103Z namespaces/kserve-ci-e2e-test/apps/replicasets.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.451420141Z namespaces/kserve-ci-e2e-test/apps/statefulsets.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.451539915Z namespaces/kserve-ci-e2e-test/autoscaling/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.451578276Z namespaces/kserve-ci-e2e-test/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.451628117Z namespaces/kserve-ci-e2e-test/batch/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.451667098Z namespaces/kserve-ci-e2e-test/batch/cronjobs.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.451750881Z namespaces/kserve-ci-e2e-test/batch/jobs.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.451793312Z namespaces/kserve-ci-e2e-test/build.openshift.io/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.451840094Z namespaces/kserve-ci-e2e-test/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.451919096Z namespaces/kserve-ci-e2e-test/build.openshift.io/builds.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.451963587Z namespaces/kserve-ci-e2e-test/core/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.452005679Z namespaces/kserve-ci-e2e-test/core/configmaps.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.452140063Z namespaces/kserve-ci-e2e-test/core/endpoints.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.452289857Z namespaces/kserve-ci-e2e-test/core/events.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.457375089Z namespaces/kserve-ci-e2e-test/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.457461901Z namespaces/kserve-ci-e2e-test/core/pods.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.457995327Z namespaces/kserve-ci-e2e-test/core/replicationcontrollers.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.458104641Z namespaces/kserve-ci-e2e-test/core/secrets.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.45842726Z namespaces/kserve-ci-e2e-test/core/services.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.458573964Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.458611525Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.458723089Z namespaces/kserve-ci-e2e-test/image.openshift.io/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.45876357Z namespaces/kserve-ci-e2e-test/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.458809531Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.458847112Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.458932645Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.458977506Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.459015917Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.459074699Z namespaces/kserve-ci-e2e-test/networking.k8s.io/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.459117451Z namespaces/kserve-ci-e2e-test/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.459160212Z namespaces/kserve-ci-e2e-test/pods/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.459167352Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.459206773Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.459319806Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1/main/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.459329837Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1/main/main/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.459333837Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1/main/main/logs/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.459353987Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1/main/main/logs/current.insecure.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.45942224Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1/main/main/logs/current.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.459498282Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1/main/main/logs/previous.insecure.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.459569634Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1/main/main/logs/previous.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.459606515Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.459646346Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.459731069Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0/main/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.459738649Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0/main/main/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.459742709Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0/main/main/logs/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.45977512Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0/main/main/logs/current.insecure.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.459844452Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0/main/main/logs/current.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.459909914Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0/main/main/logs/previous.insecure.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.459982956Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0/main/main/logs/previous.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.460014327Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.460046608Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.460152682Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f/main/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.460160432Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f/main/main/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.460164572Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f/main/main/logs/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.460197833Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f/main/main/logs/current.insecure.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.460301776Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f/main/main/logs/current.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.460375038Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f/main/main/logs/previous.insecure.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.4604444Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f/main/main/logs/previous.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.460482471Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.460521153Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.460663467Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz/llm-d-routing-sidecar/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.460670867Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz/llm-d-routing-sidecar/llm-d-routing-sidecar/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.460675017Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.460705408Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/current.insecure.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.46078426Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/current.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.460851272Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/previous.insecure.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.460922834Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/previous.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.460956115Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz/main/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.460963946Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz/main/main/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.460968116Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz/main/main/logs/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.460987326Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz/main/main/logs/current.insecure.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.461064519Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz/main/main/logs/current.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.461133561Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz/main/main/logs/previous.insecure.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.461204853Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz/main/main/logs/previous.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.461256204Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.461299016Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.461397819Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt/main/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.461407429Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt/main/main/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.461419889Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt/main/main/logs/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.46144368Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt/main/main/logs/current.insecure.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.461516522Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt/main/main/logs/current.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.461583184Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt/main/main/logs/previous.insecure.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.461663696Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt/main/main/logs/previous.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.461699828Z namespaces/kserve-ci-e2e-test/pods/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.461740759Z namespaces/kserve-ci-e2e-test/pods/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.461839692Z namespaces/kserve-ci-e2e-test/pods/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2/main/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.461847512Z namespaces/kserve-ci-e2e-test/pods/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2/main/main/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.461851572Z namespaces/kserve-ci-e2e-test/pods/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2/main/main/logs/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.461883263Z namespaces/kserve-ci-e2e-test/pods/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2/main/main/logs/current.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.462065868Z namespaces/kserve-ci-e2e-test/pods/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2/main/main/logs/previous.insecure.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.462147531Z namespaces/kserve-ci-e2e-test/pods/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelqsd2/main/main/logs/previous.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.462197132Z namespaces/kserve-ci-e2e-test/pods/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.462284485Z namespaces/kserve-ci-e2e-test/pods/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.462382568Z namespaces/kserve-ci-e2e-test/pods/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff/main/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.462405638Z namespaces/kserve-ci-e2e-test/pods/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff/main/main/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.462409939Z namespaces/kserve-ci-e2e-test/pods/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff/main/main/logs/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.46244502Z namespaces/kserve-ci-e2e-test/pods/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff/main/main/logs/current.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.462619585Z namespaces/kserve-ci-e2e-test/pods/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff/main/main/logs/previous.insecure.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.462707398Z namespaces/kserve-ci-e2e-test/pods/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schemgcff/main/main/logs/previous.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.462750179Z namespaces/kserve-ci-e2e-test/pods/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.4627959Z namespaces/kserve-ci-e2e-test/pods/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.462885073Z namespaces/kserve-ci-e2e-test/pods/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb/main/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.462897243Z namespaces/kserve-ci-e2e-test/pods/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb/main/main/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.462901493Z namespaces/kserve-ci-e2e-test/pods/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb/main/main/logs/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.462941465Z namespaces/kserve-ci-e2e-test/pods/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb/main/main/logs/current.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.463094339Z namespaces/kserve-ci-e2e-test/pods/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb/main/main/logs/previous.insecure.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.463173832Z namespaces/kserve-ci-e2e-test/pods/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehlrzb/main/main/logs/previous.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.463218413Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-r2bjb/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.463293405Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-r2bjb/router-gateway-1-openshift-default-54c47cdfc7-r2bjb.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.463397848Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-r2bjb/istio-proxy/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.463408088Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-r2bjb/istio-proxy/istio-proxy/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.463412499Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-r2bjb/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.46343994Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-r2bjb/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.463567263Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-r2bjb/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.463650896Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-c9txx/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.463689057Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-c9txx/router-gateway-2-openshift-default-684dc9b4b9-c9txx.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.4637837Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-c9txx/istio-proxy/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.46379381Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-c9txx/istio-proxy/istio-proxy/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.46379788Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-c9txx/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.463825361Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-c9txx/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.463946204Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-c9txx/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.464026807Z namespaces/kserve-ci-e2e-test/policy/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.464071798Z namespaces/kserve-ci-e2e-test/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.464111989Z namespaces/kserve-ci-e2e-test/route.openshift.io/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.464155961Z namespaces/kserve-ci-e2e-test/route.openshift.io/routes.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.464197832Z namespaces/kuadrant-system/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.464268894Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.464343967Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.464390608Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.464430819Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.46447694Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.464555383Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.464896313Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.465177251Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.465220982Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.465288644Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.465326596Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.465373607Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.465450729Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.465498511Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.465533802Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.465615594Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.465666016Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.465705107Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.465898563Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.466253143Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.466712717Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.4667979Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.46716414Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.467313335Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.467525721Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.467642255Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.467677686Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.467789339Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.46783003Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.467874122Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.467914203Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.467997685Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.468041006Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.468082228Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.468127759Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.46816776Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.468214022Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.468220842Z namespaces/kuadrant-system/pods/authorino-7c77957575-cfthr/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.468285674Z namespaces/kuadrant-system/pods/authorino-7c77957575-cfthr/authorino-7c77957575-cfthr.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.468366266Z namespaces/kuadrant-system/pods/authorino-7c77957575-cfthr/authorino/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.468376076Z namespaces/kuadrant-system/pods/authorino-7c77957575-cfthr/authorino/authorino/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.468380277Z namespaces/kuadrant-system/pods/authorino-7c77957575-cfthr/authorino/authorino/logs/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.468420558Z namespaces/kuadrant-system/pods/authorino-7c77957575-cfthr/authorino/authorino/logs/current.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.471986094Z namespaces/kuadrant-system/pods/authorino-7c77957575-cfthr/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.472064337Z namespaces/kuadrant-system/pods/authorino-7c77957575-cfthr/authorino/authorino/logs/previous.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.472119618Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xdn96/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.472157269Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xdn96/authorino-operator-6d85f6564-xdn96.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.472311034Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xdn96/manager/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.472350355Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xdn96/manager/manager/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.472355575Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xdn96/manager/manager/logs/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.472374286Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xdn96/manager/manager/logs/current.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.472588442Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xdn96/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.472667935Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xdn96/manager/manager/logs/previous.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.472705446Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-hphfv/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.472764687Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-hphfv/dns-operator-controller-manager-65b49595d7-hphfv.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.47285933Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-hphfv/manager/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.472866661Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-hphfv/manager/manager/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.47287045Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-hphfv/manager/manager/logs/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.472906192Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-hphfv/manager/manager/logs/current.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.473032785Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-hphfv/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.473108068Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-hphfv/manager/manager/logs/previous.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.473157379Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-4tc92/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.47320194Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-4tc92/kuadrant-console-plugin-85fd4b8b9d-4tc92.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.473321784Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-4tc92/kuadrant-console-plugin/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.473334814Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-4tc92/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.473339104Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-4tc92/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.473364645Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-4tc92/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.473442398Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-4tc92/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.47351314Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-4tc92/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.473560521Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.473606663Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.473718826Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x/manager/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.473729316Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x/manager/manager/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.473740147Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x/manager/manager/logs/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.473762587Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x/manager/manager/logs/current.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.487027683Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.487087164Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x/manager/manager/logs/previous.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.487132496Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-hnvp6/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.487175037Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-hnvp6/limitador-limitador-69574b596d-hnvp6.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.487322631Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-hnvp6/limitador/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.487335942Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-hnvp6/limitador/limitador/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.487340872Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-hnvp6/limitador/limitador/logs/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.487365563Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-hnvp6/limitador/limitador/logs/current.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.487453805Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-hnvp6/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.487522357Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-hnvp6/limitador/limitador/logs/previous.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.487560568Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-mw8p8/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.48760746Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-mw8p8/limitador-operator-controller-manager-6f9f468797-mw8p8.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.487706433Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-mw8p8/manager/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.487714703Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-mw8p8/manager/manager/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.487718683Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-mw8p8/manager/manager/logs/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.487749624Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-mw8p8/manager/manager/logs/current.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.487900569Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-mw8p8/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.487968541Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-mw8p8/manager/manager/logs/previous.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.487996692Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.488038053Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.488092714Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.488138436Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.488179987Z namespaces/openshift-ingress/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.488236359Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.488331002Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.488365943Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.488414074Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.488460705Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.488547498Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.488764114Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.48896296Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.489005752Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.489049493Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.489108305Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.489146146Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.489253049Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.48929986Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.489343472Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.489421964Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.489465845Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.489510067Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.489991691Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.490093724Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.490420304Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.490493806Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.490749403Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.490875777Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.491062533Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.491150226Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.491181367Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.491286639Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.49130838Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.491369382Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.491406823Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.491498506Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.491536777Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.491577768Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.49163307Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.491666161Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.491714192Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.491725052Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zprl6/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.491758284Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zprl6/istiod-openshift-gateway-94bb8fbfd-zprl6.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.491859927Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zprl6/discovery/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.491873147Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zprl6/discovery/discovery/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.491878047Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zprl6/discovery/discovery/logs/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.4919645Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zprl6/discovery/discovery/logs/current.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.522270653Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zprl6/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.522351456Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zprl6/discovery/discovery/logs/previous.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.522385287Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-8g97n/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.522435688Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-8g97n/openshift-ai-inference-openshift-default-655f7c58b4-8g97n.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.522567012Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-8g97n/istio-proxy/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.522575872Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-8g97n/istio-proxy/istio-proxy/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.522580112Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-8g97n/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.522616164Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-8g97n/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.523665795Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-8g97n/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.523949653Z namespaces/openshift-ingress/pods/router-default-586b57644b-q24lm/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.523994905Z namespaces/openshift-ingress/pods/router-default-586b57644b-q24lm/router-default-586b57644b-q24lm.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.524131729Z namespaces/openshift-ingress/pods/router-default-586b57644b-q24lm/router/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.524139059Z namespaces/openshift-ingress/pods/router-default-586b57644b-q24lm/router/router/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.524143049Z namespaces/openshift-ingress/pods/router-default-586b57644b-q24lm/router/router/logs/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.52418286Z namespaces/openshift-ingress/pods/router-default-586b57644b-q24lm/router/router/logs/current.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.524352995Z namespaces/openshift-ingress/pods/router-default-586b57644b-q24lm/router/router/logs/previous.insecure.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.524438318Z namespaces/openshift-ingress/pods/router-default-586b57644b-q24lm/router/router/logs/previous.log [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.524485589Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.52453071Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.524591902Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.524647274Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.535384134Z [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.535404985Z sent 3,917 bytes received 1,325,716 bytes 2,659,266.00 bytes/sec [must-gather] [must-gather-wkpst] OUT 2026-07-13T20:24:35.535411005Z total size is 18,678,210 speedup is 14.05 [must-gather] [must-gather ] OUT 2026-07-13T20:24:35.801342282Z namespace/openshift-must-gather-stwlp deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 79ef62dc-8bf7-4a50-8cc9-d8dfbff73685 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.24" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-13T20:24:35.930930745Z Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ec435553f79197f28a94d55da851b8302383238997d790ed3335149e3ee4341f [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 79ef62dc-8bf7-4a50-8cc9-d8dfbff73685 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.24" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-13T20:24:35.96262766Z namespace/openshift-must-gather-s6wcw created [must-gather] [must-gather ] OUT 2026-07-13T20:24:35.982348868Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-vtqrb created [must-gather] [must-gather ] OUT 2026-07-13T20:24:36.068618779Z pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ec435553f79197f28a94d55da851b8302383238997d790ed3335149e3ee4341f created [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:37.450721175Z [disk usage checker] Started [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:37.454449814Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:37.687450117Z Gathering data for ns/openshift-cluster-version... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:37.942753478Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:38.200178482Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:38.381189440Z Gathering data for ns/default... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:38.901809542Z Gathering data for ns/openshift... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:39.507045612Z Gathering data for ns/kube-system... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:40.675218264Z Waiting on subprocesses to finish execution. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:40.684552000Z INFO: Gathering machine config daemon's old logs from all nodes [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:40.685364145Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:40.686656844Z INFO: Collecting host service logs for crio [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:40.686981957Z INFO: Collecting host service logs for kubelet [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:40.687526004Z INFO: Collecting host service logs for rpm-ostreed [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:40.687990280Z INFO: Collecting host service logs for ostree-finalize-staged [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:40.688318076Z INFO: Collecting host service logs for machine-config-daemon-firstboot [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:40.688614729Z INFO: Collecting host service logs for machine-config-daemon-host [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:40.688934753Z INFO: Collecting host service logs for NetworkManager [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:40.690413419Z INFO: Collecting host service logs for openvswitch [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:40.690768648Z INFO: Collecting host service logs for ovs-configuration [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:40.691057004Z INFO: Collecting host service logs for ovsdb-server [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:40.691372892Z INFO: Collecting host service logs for ovs-vswitchd [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:40.691734037Z INFO: Waiting for worker host service log collection to complete ... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:40.692149458Z INFO: Gathering on-disk MachineConfig from degraded nodes [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:40.692724500Z INFO: Gathering HAProxy config files [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:40.735041149Z Executing Istio gather script [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:40.784635681Z INFO: Waiting for node performance related collection to complete ... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:41.325148106Z Gathering data for ns/openshift-etcd... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:41.506429848Z No resources found in openshift-etcd namespace. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:41.956284187Z INFO: Collecting Insights Archives from insights-operator-68d9dd94cf-tbqhd [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:41.956375182Z insights-runtime-extractor-2h9jt [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:41.956397072Z insights-runtime-extractor-7kmgq [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:41.956417479Z insights-runtime-extractor-xmzjq [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:42.097442863Z INFO: Waiting for HAProxy config collection to complete ... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:42.123971045Z Wrote inspect data to must-gather. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:42.326016530Z ERROR: No running kube-apiserver pods found [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:42.339531637Z INFO: namespace openshift-frr-k8s not detected. Skipping. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:42.350648721Z INFO: "sriov-network-operator" not detected. Skipping. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:42.381526678Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:42.381526678Z See 'oc rsync -h' for help and examples [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:42.398803580Z INFO: Found 1 replicas - prometheus-k8s-0 [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:42.687900914Z No resources found [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:42.702314666Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:42.781540010Z INFO: "metallb-operator" not detected. Skipping. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:42.829256393Z INFO: Waiting for on-disk MachineConfig collection to complete ... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:42.829355290Z INFO: on-disk MachineConfig config collection complete. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:42.841441072Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:42.969257577Z error: the server doesn't have a resource type "performanceprofile" [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:43.206301789Z INFO: Worker host service log collection to complete. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:43.213578785Z Wrote inspect data to must-gather. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:43.366225481Z Wrote inspect data to must-gather. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:43.405265360Z No resources found [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:43.586727989Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:43.659256866Z Wrote inspect data to must-gather. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:43.659314053Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:43.659314053Z namespaces "assisted-installer" not found [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:43.691402733Z INFO: 'previous-logs' folder not found on ip-10-0-128-215.ec2.internal, skipping... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:43.922615195Z INFO: Getting alertmanagers from prometheus-k8s-0 [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:43.944501103Z tar: Removing leading `/' from member names [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:43.949578791Z INFO: HAProxy config collection complete. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:44.066712813Z error: the server doesn't have a resource type "multi-networkpolicy" [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:44.141732591Z Wrote inspect data to must-gather. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:44.211563053Z Wrote inspect data to must-gather. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:44.322710563Z Wrote inspect data to must-gather. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:44.350503751Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:44.434305897Z INFO: 'previous-logs' folder not found on ip-10-0-131-203.ec2.internal, skipping... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:44.643812065Z error: the server doesn't have a resource type "machineconfigs" [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:44.817264079Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:44.828829005Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:44.850109374Z INFO: Getting rules from prometheus-k8s-0 [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:44.857725101Z Wrote inspect data to must-gather. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:44.858933342Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:45.089364164Z error: the server doesn't have a resource type "machineconfigpools" [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:45.126210918Z INFO: 'previous-logs' folder not found on ip-10-0-139-134.ec2.internal, skipping... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:45.126341317Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:45.126399500Z INFO: Machine Config Daemon termination log collection complete. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:45.334449146Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:45.338765208Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:45.467339973Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:45.474600384Z Wrote inspect data to must-gather. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:45.493172333Z Wrote inspect data to must-gather. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:45.691924753Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:45.700063954Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:45.746406651Z INFO: Getting status/config from prometheus-k8s-0 [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:45.799462528Z error: the server doesn't have a resource type "kubeletconfigs" [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:45.961026376Z Wrote inspect data to must-gather. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:46.052587700Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:46.084077774Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:46.087679622Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:46.193061613Z Wrote inspect data to must-gather. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:46.400091715Z INFO: INTERCONNECT MODE [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:46.401653948Z INFO: Gathering ovn-kubernetes DBs [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:46.456816629Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:46.462809620Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:46.510913934Z INFO: Getting status/flags from prometheus-k8s-0 [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:46.522173118Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:46.605577738Z Wrote inspect data to must-gather. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:46.670223757Z INFO: Gathering OVN_Northbound from ovnkube-node-2bscc... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:46.670580916Z INFO: Gathering OVN_Northbound from ovnkube-node-c2s8k... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:46.670856406Z INFO: Gathering OVN_Northbound from ovnkube-node-v76kh... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:46.671175728Z INFO: Gathering OVN_Southbound from ovnkube-node-2bscc... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:46.671484151Z INFO: Gathering OVN_Southbound from ovnkube-node-c2s8k... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:46.672280619Z INFO: Gathering OVN_Southbound from ovnkube-node-v76kh... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:46.873470715Z Gathering data for ns/openshift-monitoring... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:46.966921460Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:46.990191975Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:47.103271047Z Wrote inspect data to must-gather. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:47.254860755Z tar: Removing leading `/' from member names [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:47.256431951Z tar: Removing leading `/' from member names [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:47.340211339Z tar: Removing leading `/' from member names [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:47.499501417Z tar: Removing leading `/' from member names [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:47.518850049Z Wrote inspect data to must-gather. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:47.556614888Z tar: Removing leading `/' from member names [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:47.579886206Z tar: Removing leading `/' from member names [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:47.639542470Z INFO: Getting targets?state=active from prometheus-k8s-0 [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:47.774437829Z Wrote inspect data to must-gather. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:47.779903375Z INFO: Waiting for network log collection to complete ... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:47.779956485Z INFO: Waiting for ovnk database copies to complete ... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:47.780459029Z INFO: Copying ovnk databases complete. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:47.782527351Z 7.4M must-gather/network_logs/ovnk_database_store [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:47.785257261Z ovnk_database_store/ [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:47.785257261Z ovnk_database_store/ovnkube-node-c2s8k_nbdb [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:47.802163026Z ovnk_database_store/ovnkube-node-2bscc_sbdb [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:47.844928244Z ovnk_database_store/ovnkube-node-2bscc_nbdb [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:47.854204066Z Wrote inspect data to must-gather. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:47.857547437Z ovnk_database_store/ovnkube-node-c2s8k_sbdb [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:47.901188356Z ovnk_database_store/ovnkube-node-v76kh_sbdb [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:47.904951315Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:47.944793959Z ovnk_database_store/ovnkube-node-v76kh_nbdb [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:47.963929884Z INFO: Network log collection complete. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:48.101809976Z Wrote inspect data to must-gather. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:48.204114020Z INFO: Getting status/tsdb from prometheus-k8s-0 [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:48.358328532Z Wrote inspect data to must-gather. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:48.631308873Z INFO: Getting status from alertmanager-main-0 [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:48.650081562Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:48.652242096Z INFO: Fallback to identify the container image from release info [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:49.353566087Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:49.507473298Z daemonset.apps/perf-node-gather-daemonset created [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:49.622253118Z Inspecting resource crd/authorizationpolicies.security.istio.io [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:49.660453334Z Waiting for performance profile collector pods to become ready: 1 [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:49.808072645Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:50.030363040Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:50.032786730Z Inspecting resource crd/destinationrules.networking.istio.io [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:50.304208921Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:50.544102856Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:50.546431165Z Inspecting resource crd/envoyfilters.networking.istio.io [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:50.748580937Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:50.810739256Z Waiting for performance profile collector pods to become ready: 2 [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:50.992000278Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:50.995733154Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:51.059618961Z Gathering data for ns/openshift-network-console... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:51.207020130Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:51.434492482Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:51.437625472Z Inspecting resource crd/gateways.gateway.networking.k8s.io [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:51.644621790Z Gathering data for ns/openshift-console-operator... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:51.665087538Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:51.910134381Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:51.912966521Z Inspecting resource crd/gateways.networking.istio.io [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:51.962499891Z Waiting for performance profile collector pods to become ready: 3 [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:52.138083722Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:52.158474667Z Gathering data for ns/openshift-console... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:52.370572153Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:52.373102002Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:52.579387333Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:52.825925332Z Gathering data for ns/openshift-cluster-storage-operator... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:52.826126258Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:52.828799182Z Inspecting resource crd/httproutes.gateway.networking.k8s.io [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:52.911954658Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:53.084136140Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:53.103716997Z Waiting for performance profile collector pods to become ready: 4 [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:53.336820039Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:53.339711256Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:53.480337010Z Gathering data for ns/openshift-dns-operator... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:53.552023384Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:53.782705117Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:53.785781352Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:53.906721920Z Gathering data for ns/openshift-dns... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:53.980676839Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:54.191562398Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:54.194336837Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:54.240283906Z Waiting for performance profile collector pods to become ready: 5 [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:54.394047608Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:54.624380304Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:54.627411016Z Inspecting resource crd/inferencepools.inference.networking.k8s.io [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:54.723909828Z Gathering data for ns/openshift-image-registry... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:54.819018017Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:55.041598468Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:55.044638639Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:55.227756349Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:55.405544852Z Waiting for performance profile collector pods to become ready: 6 [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:55.407647165Z Gathering data for ns/openshift-ingress-operator... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:55.491681661Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:55.495805759Z Inspecting resource crd/peerauthentications.security.istio.io [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:55.679770950Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:55.832531950Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:55.897269962Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:55.899807196Z Inspecting resource crd/proxyconfigs.networking.istio.io [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:56.071941127Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:56.265117418Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:56.267708853Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:56.443984045Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:56.548838840Z Waiting for performance profile collector pods to become ready: 7 [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:56.672365962Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:56.676106684Z Inspecting resource crd/requestauthentications.security.istio.io [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:56.864101703Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:57.101969409Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:57.106918167Z Inspecting resource crd/serviceentries.networking.istio.io [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:57.140920453Z Gathering data for ns/openshift-ingress-canary... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:57.311016091Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:57.530828931Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:57.534897529Z Inspecting resource crd/sidecars.networking.istio.io [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:57.666212850Z Gathering data for ns/openshift-insights... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:57.708952469Z Daemonset perf-node-gather-daemonset ready 3 out of 3 [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:57.780785461Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:57.917639342Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:58.001402140Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:58.004248886Z Inspecting resource crd/telemetries.telemetry.istio.io [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:58.008639742Z Collecting performance related data for node ip-10-0-128-215.ec2.internal [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:58.016592702Z Collecting performance related data for node ip-10-0-139-134.ec2.internal [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:58.024886876Z Collecting performance related data for node ip-10-0-131-203.ec2.internal [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:58.247910567Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:58.502662910Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:58.507166794Z Inspecting resource crd/virtualservices.networking.istio.io [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:58.802948592Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:59.037141387Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:59.039996023Z Inspecting resource crd/wasmplugins.extensions.istio.io [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:59.291785153Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:59.550594517Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:59.554121376Z Inspecting resource crd/workloadentries.networking.istio.io [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:24:59.788265810Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:00.010574623Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:00.012751573Z Inspecting resource crd/workloadgroups.networking.istio.io [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:00.208026336Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:00.410935770Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:00.576306560Z Inspecting resource mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector-openshift-gateway-openshift-ingress [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:00.837381463Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:01.273985544Z Gathering data for ns/openshift-lws-operator... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:01.899142569Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:02.075626933Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:02.222223870Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:02.349536507Z Inspecting resource validatingwebhookconfiguration.admissionregistration.k8s.io/istio-validator-openshift-gateway-openshift-ingress [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:02.363900204Z Gathering data for ns/kserve... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:02.572512784Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:02.923178341Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:03.764825766Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:03.948498493Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:04.117435558Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:04.617003279Z Done executing Istio gather script [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:05.214929152Z Gathering data for ns/openshift-config... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:05.574282912Z Gathering data for ns/openshift-config-managed... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:06.171237961Z Gathering data for ns/openshift-kube-apiserver-operator... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:06.623710602Z Gathering data for ns/openshift-kube-apiserver... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:06.847657957Z Collecting kubelet logs for node ip-10-0-128-215.ec2.internal [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:06.849652286Z Collecting kubelet logs for node ip-10-0-139-134.ec2.internal [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:06.851467697Z Collecting kubelet logs for node ip-10-0-131-203.ec2.internal [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:07.001420532Z Gathering data for ns/openshift-kube-controller-manager... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:07.338177089Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-s6wcw namespace [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:07.343148851Z INFO: Node performance data collection complete. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:07.526008442Z Gathering data for ns/openshift-kube-controller-manager-operator... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:07.929964561Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:07.983473282Z Gathering data for ns/openshift-kube-scheduler... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:08.330720080Z Gathering data for ns/openshift-kube-scheduler-operator... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:08.805981569Z Gathering data for ns/openshift-kube-storage-version-migrator... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:09.278817338Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:09.807193685Z Gathering data for ns/openshift-user-workload-monitoring... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:10.631116976Z Gathering data for ns/openshift-multus... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:12.937271180Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:13.232742292Z Gathering data for ns/openshift-ovn-kubernetes... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:15.127490123Z Gathering data for ns/openshift-host-network... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:15.497632308Z Gathering data for ns/openshift-network-diagnostics... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:16.306830482Z Gathering data for ns/openshift-network-node-identity... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:16.856364806Z Gathering data for ns/openshift-network-operator... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:17.709468380Z Gathering data for ns/openshift-cloud-network-config-controller... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:17.944327931Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:18.030453667Z Gathering data for ns/openshift-cluster-node-tuning-operator... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:18.644106851Z Gathering data for ns/openshift-apiserver-operator... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:19.005587241Z Gathering data for ns/openshift-apiserver... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:19.491337321Z Gathering data for ns/openshift-controller-manager-operator... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:19.806556342Z Gathering data for ns/openshift-controller-manager... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:20.144468446Z Gathering data for ns/openshift-cluster-samples-operator... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:20.895506621Z Gathering data for ns/openshift-operator-lifecycle-manager... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:21.311112676Z Gathering data for ns/openshift-service-ca-operator... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:21.690148540Z Gathering data for ns/openshift-service-ca... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:22.172643322Z Gathering data for ns/openshift-cluster-csi-drivers... [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:22.949126442Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:23.253666478Z Wrote inspect data to must-gather. [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:23.253714155Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:23.253714155Z skipping gathering secrets/support due to error: secrets "support" not found [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:23.393447149Z error: the server doesn't have a resource type "clusters" [must-gather] [must-gather-b2w4x] POD 2026-07-13T20:25:23.541172228Z Caches written to disk [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.552127046Z waiting for gather to complete [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.556557868Z downloading gather output [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.888967307Z receiving incremental file list [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.921173478Z ./ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.921297022Z aggregated-discovery-api.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.921633082Z aggregated-discovery-apis.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.922409235Z event-filter.html [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.92525213Z timestamp [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.925366284Z version [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.934853707Z cluster-scoped-resources/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.934886788Z cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.934903748Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.934938799Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/default-network-annotation.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.935103644Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/servicecidrs.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.93529954Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/user-defined-networks-namespace-label.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.935332701Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.935357522Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/default-network-annotation-binding.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.935563068Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/servicecidrs-binding.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.935707222Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/user-defined-networks-namespace-label-binding.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.935730963Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.935740093Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/multus.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.935855817Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/network-node-identity.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.935921109Z cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.936387712Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.936428644Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/accounts.nim.opendatahub.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.93664003Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.936913388Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminpolicybasedexternalroutes.k8s.ovn.org.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.937113334Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertingrules.monitoring.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.937279659Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagerconfigs.monitoring.coreos.com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.941281859Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagers.monitoring.coreos.com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.94300894Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertrelabelconfigs.monitoring.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.943132834Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyapprovals.devportal.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.943246877Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyrequests.devportal.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.943384191Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeys.devportal.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.943519735Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiproducts.devportal.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.94401672Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apirequestcounts.apiserver.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.944145094Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiservers.config.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.944320929Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/appliedmanifestworks.work.open-cluster-management.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.944444243Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authconfigs.authorino.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.945276208Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authentications.config.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.945545926Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorinos.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.945704181Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.945866906Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authpolicies.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.947003499Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/baselineadminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.947256427Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/builds.config.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.947429592Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/bundles.trust.cert-manager.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.947556666Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/catalogsources.operators.coreos.com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.947832224Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificaterequests.cert-manager.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.947977668Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificates.cert-manager.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.948166304Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certmanagers.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.948464783Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/challenges.acme.cert-manager.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.949206025Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudcredentials.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.949441462Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.949474733Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudprivateipconfigs.cloud.network.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.949545085Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterclaims.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.949652108Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.949752081Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercsidrivers.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.949957577Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterimagepolicies.config.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.950138323Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterissuers.cert-manager.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.950909966Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteroperators.config.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.95105113Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterresourcequotas.quota.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.951173864Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterserviceversions.operators.coreos.com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.953136442Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterstoragecontainers.serving.kserve.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.953314998Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustertriggerauthentications.keda.sh.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.953475993Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteruserdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.9537142Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterversions.config.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.953999998Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.954439321Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.954547694Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.samples.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.954673688Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleclidownloads.console.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.954780631Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleexternalloglinks.console.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.954892005Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolelinks.console.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.955015218Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolenotifications.console.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.955126872Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleplugins.console.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.955293797Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolequickstarts.console.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.955430121Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.config.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.955540004Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.955791851Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolesamples.console.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.955955727Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleyamlsamples.console.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.95606771Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/controllerconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.956637627Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/credentialsrequests.cloudcredential.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.9567568Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/csisnapshotcontrollers.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.956860103Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/datascienceclusters.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.957042589Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.957853033Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.config.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.957965687Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.958160362Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnshealthcheckprobes.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.958283946Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnspolicies.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.958449921Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.ingress.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.958564834Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.958721419Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dscinitializations.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.958851973Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressfirewalls.k8s.ovn.org.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.958967957Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressips.k8s.ovn.org.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.95908238Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressqoses.k8s.ovn.org.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.959203793Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressrouters.network.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.959355728Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressservices.k8s.ovn.org.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.959481702Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.959640157Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/etcds.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.95975459Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/featuregates.config.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.959897094Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.960044069Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.96042846Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.960652147Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.961023798Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/helmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.961135431Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.961941975Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httpscaledobjects.http.keda.sh.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.962072369Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentpolicies.config.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.962188902Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentsourcepolicies.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.962323717Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagedigestmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.96244126Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepolicies.config.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.962616645Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepruners.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.962907464Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/images.config.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.963014497Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagetagmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.963138831Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencegraphs.serving.kserve.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.963315986Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.9634393Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.963554403Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepoolimports.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.963701018Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.963856312Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.963973526Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.966199122Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/infrastructures.config.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.966649306Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresscontrollers.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.967212712Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresses.config.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.967397378Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/insightsoperators.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.967556283Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/installplans.operators.coreos.com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.967688406Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/interceptorroutes.http.keda.sh.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.96781931Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ipamclaims.k8s.cni.cncf.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.967927764Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ippools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.968032767Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/issuers.cert-manager.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.968877112Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiocsrs.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.969187531Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kedacontrollers.keda.sh.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.971986165Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kuadrants.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.972110109Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeapiservers.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.972254863Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubecontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.97484214Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeschedulers.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.974996175Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubestorageversionmigrators.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.975135619Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersetoperators.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.975297014Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersets.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.977925172Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/limitadors.limitador.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.97819619Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceserviceconfigs.serving.kserve.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.985009483Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.989089045Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfignodes.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.98925005Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfigurations.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.989531249Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosbuilds.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.989697623Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.989842018Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/network-attachment-definitions.k8s.cni.cncf.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.989946601Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.config.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.990106335Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.990370783Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodes.config.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.990489357Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodeslicepools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.99059343Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oauths.config.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.990776175Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oidcpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.990905429Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/olmconfigs.operators.coreos.com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.991009813Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftapiservers.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.991120546Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftcontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.991243849Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorconditions.operators.coreos.com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.991382033Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorgroups.operators.coreos.com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.991521608Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorhubs.config.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.991653642Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorpkis.network.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.991766555Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operators.operators.coreos.com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.991907699Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/orders.acme.cert-manager.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.992051593Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/overlappingrangeipreservations.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.992153277Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.992301631Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/pinnedimagesets.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.992423015Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/planpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.992545968Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podmonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.992811976Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podnetworkconnectivitychecks.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.99293447Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/probes.monitoring.coreos.com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.993177237Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/profiles.tuned.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.993305221Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projecthelmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.993422414Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projects.config.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.993523648Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheuses.monitoring.coreos.com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.995424534Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheusrules.monitoring.coreos.com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.995547828Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxies.config.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.995660421Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.995772435Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rangeallocations.security.internal.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.995868978Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.996018852Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.996135405Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.99630656Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rolebindingrestrictions.authorization.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.996435184Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledjobs.keda.sh.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.997997071Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledobjects.keda.sh.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.998147026Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/schedulers.config.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.998278059Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/securitycontextconstraints.security.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.998426224Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicecas.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.998535927Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.998714012Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicemonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.99895616Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servingruntimes.serving.kserve.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.999476635Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.999761384Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storages.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.999875147Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storagestates.migration.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:24.99997881Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storageversionmigrations.migration.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.000094094Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/subscriptions.operators.coreos.com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.000715972Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.000904038Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetrypolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.001016891Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/thanosrulers.monitoring.coreos.com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.002428733Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tlspolicies.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.002574407Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tokenratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.002723482Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trainedmodels.serving.kserve.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.002829585Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/triggerauthentications.keda.sh.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.00297968Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trustmanagers.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.003245058Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tuneds.tuned.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.003382192Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/userdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.003536556Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.003915928Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumepopulators.populator.storage.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.004016861Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotclasses.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.0059988Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotcontents.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.006208616Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshots.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.006402512Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.006550116Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.006721722Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.006875946Z cluster-scoped-resources/apiregistration.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.007173765Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.007197846Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1..yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.007317979Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.acme.cert-manager.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.007405962Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.007477884Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.007567347Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiserver.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.007714251Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.007859975Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.007987239Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.008116933Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.008381111Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.008511155Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.autoscaling.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.008591577Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.batch.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.00868553Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.build.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.008775172Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cert-manager.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.008860305Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.008949058Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloud.network.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.00903356Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloudcredential.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.009122473Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.config.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.009208456Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.console.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.009316419Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.009403531Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.009491254Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.discovery.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.009579257Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.009669569Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.events.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.009754612Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.flowcontrol.apiserver.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.009841344Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.009931027Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.image.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.01001878Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.010102692Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.inference.networking.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.010187695Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.ingress.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.010307718Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.010405971Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.ovn.org.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.010490844Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.010574926Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.010661539Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.010760462Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.coreos.com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.010861575Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.010937407Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.network.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.01102667Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.011112682Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.011198055Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.nim.opendatahub.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.011325079Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.node.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.011423202Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.oauth.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.011510694Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.011614967Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operators.coreos.com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.01169577Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.packages.operators.coreos.com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.011786553Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.policy.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.011898616Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.project.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.011984859Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.quota.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.012107422Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.012257287Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.resource.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.012393941Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.route.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.012528365Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.samples.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.012638078Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.scheduling.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.012751961Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.internal.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.012874955Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.012994298Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.013126753Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.013271077Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.storage.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.01339069Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.telemetry.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.013516024Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.template.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.013645278Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.tuned.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.013771522Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.user.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.013903225Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.work.open-cluster-management.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.0140629Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.014194904Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.014341399Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.devportal.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.014476653Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.eventing.keda.sh.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.014609367Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.014737921Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.014864824Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.http.keda.sh.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.015003178Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.015155733Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.015279997Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.keda.sh.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.01538619Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.015472852Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.limitador.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.015583846Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.migration.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.015664738Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.monitoring.coreos.com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.015762521Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operator.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.015851264Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operators.coreos.com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.015941426Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.policy.networking.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.016030729Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.serving.kserve.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.016126832Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.telemetry.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.016216075Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.trust.cert-manager.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.016347378Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.016453242Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.016543415Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.operators.coreos.com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.016638197Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.serving.kserve.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.01672465Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha3.networking.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.016819933Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.016912546Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.external.metrics.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.017023519Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.017128342Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.helm.openshift.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.017217404Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.http.keda.sh.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.017343688Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.017436501Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.metrics.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.017546134Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.monitoring.coreos.com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.017632167Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.networking.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.01772864Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.017820273Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.populator.storage.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.017916096Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.security.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.018002418Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.serving.kserve.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.018100171Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.storage.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.018184504Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta3.authorino.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.018313147Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.autoscaling.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.01840324Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.operators.coreos.com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.018445221Z cluster-scoped-resources/certificates.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.018481152Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.018516833Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-6q89g.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.018659907Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-t8zzd.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.018759461Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-xtz8t.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.018944406Z cluster-scoped-resources/config.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.018984497Z cluster-scoped-resources/config.openshift.io/apiservers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.019164442Z cluster-scoped-resources/config.openshift.io/authentications.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.019330327Z cluster-scoped-resources/config.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.01941309Z cluster-scoped-resources/config.openshift.io/clusterimagepolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.019499363Z cluster-scoped-resources/config.openshift.io/clusteroperators.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.019893964Z cluster-scoped-resources/config.openshift.io/clusterversions.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.020006188Z cluster-scoped-resources/config.openshift.io/consoles.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.02008718Z cluster-scoped-resources/config.openshift.io/dnses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.020181143Z cluster-scoped-resources/config.openshift.io/featuregates.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.020328767Z cluster-scoped-resources/config.openshift.io/imagecontentpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.02041694Z cluster-scoped-resources/config.openshift.io/imagedigestmirrorsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.020504983Z cluster-scoped-resources/config.openshift.io/imagepolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.020584465Z cluster-scoped-resources/config.openshift.io/images.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.020677158Z cluster-scoped-resources/config.openshift.io/imagetagmirrorsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.02075608Z cluster-scoped-resources/config.openshift.io/infrastructures.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.020856213Z cluster-scoped-resources/config.openshift.io/ingresses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.020942016Z cluster-scoped-resources/config.openshift.io/networks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.021042249Z cluster-scoped-resources/config.openshift.io/nodes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.021125971Z cluster-scoped-resources/config.openshift.io/oauths.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.021213904Z cluster-scoped-resources/config.openshift.io/operatorhubs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.021330457Z cluster-scoped-resources/config.openshift.io/projects.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.0214115Z cluster-scoped-resources/config.openshift.io/proxies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.021493942Z cluster-scoped-resources/config.openshift.io/schedulers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.021588675Z cluster-scoped-resources/config.openshift.io/clusteroperators/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.021622036Z cluster-scoped-resources/config.openshift.io/clusteroperators/console.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.021732549Z cluster-scoped-resources/config.openshift.io/clusteroperators/csi-snapshot-controller.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.021824042Z cluster-scoped-resources/config.openshift.io/clusteroperators/dns.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.021921415Z cluster-scoped-resources/config.openshift.io/clusteroperators/image-registry.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.022003457Z cluster-scoped-resources/config.openshift.io/clusteroperators/ingress.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.02209929Z cluster-scoped-resources/config.openshift.io/clusteroperators/insights.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.022198343Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-apiserver.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.022367868Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-controller-manager.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.022460091Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-scheduler.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.022553094Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-storage-version-migrator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.022649867Z cluster-scoped-resources/config.openshift.io/clusteroperators/monitoring.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.022738599Z cluster-scoped-resources/config.openshift.io/clusteroperators/network.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.022881153Z cluster-scoped-resources/config.openshift.io/clusteroperators/node-tuning.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.022969606Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-apiserver.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.023068639Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-controller-manager.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.023154222Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-samples.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.023283766Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-catalog.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.023367188Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-packageserver.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.023467151Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.023560924Z cluster-scoped-resources/config.openshift.io/clusteroperators/service-ca.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.023656677Z cluster-scoped-resources/config.openshift.io/clusteroperators/storage.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.023714738Z cluster-scoped-resources/config.openshift.io/clusterversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.02376355Z cluster-scoped-resources/config.openshift.io/clusterversions/version.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.023827902Z cluster-scoped-resources/config.openshift.io/consoles/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.023883623Z cluster-scoped-resources/config.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.023941485Z cluster-scoped-resources/config.openshift.io/featuregates/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.023978676Z cluster-scoped-resources/config.openshift.io/featuregates/cluster.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.024072469Z cluster-scoped-resources/config.openshift.io/infrastructures/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.02410623Z cluster-scoped-resources/config.openshift.io/infrastructures/cluster.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.024180982Z cluster-scoped-resources/config.openshift.io/oauths/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.024197303Z cluster-scoped-resources/config.openshift.io/oauths/cluster.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.024294166Z cluster-scoped-resources/config.openshift.io/proxies/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.024343527Z cluster-scoped-resources/config.openshift.io/proxies/cluster.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.02443077Z cluster-scoped-resources/console.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.02444149Z cluster-scoped-resources/console.openshift.io/consoleplugins/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.024531053Z cluster-scoped-resources/console.openshift.io/consoleplugins/kuadrant-console-plugin.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.024655126Z cluster-scoped-resources/console.openshift.io/consoleplugins/monitoring-plugin.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.024758409Z cluster-scoped-resources/console.openshift.io/consoleplugins/networking-console-plugin.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.024824702Z cluster-scoped-resources/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.024833622Z cluster-scoped-resources/core/nodes/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.024888764Z cluster-scoped-resources/core/nodes/ip-10-0-128-215.ec2.internal.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.02509671Z cluster-scoped-resources/core/nodes/ip-10-0-131-203.ec2.internal.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.025251654Z cluster-scoped-resources/core/nodes/ip-10-0-139-134.ec2.internal.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.025387208Z cluster-scoped-resources/core/persistentvolumes/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.025425589Z cluster-scoped-resources/core/persistentvolumes/pvc-3823dadc-7c52-412c-8ef9-84caab00a904.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.025515052Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.025578504Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.025617905Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/catch-all.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.025716648Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/exempt.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.02579966Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/global-default.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.025886183Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-controller-manager.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.025968726Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-scheduler.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.026053468Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-system-service-accounts.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.026209693Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.026409419Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-sar.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.026502461Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.026600224Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-authentication-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.026694017Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-controller-manager.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.02677994Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.026877463Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-monitoring-metrics.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.026965835Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver-sar.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.027063968Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.027152941Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-server.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.027243314Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/probes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.027355407Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/service-accounts.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.027439769Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-leader-election.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.027526592Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-node-high.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.027630405Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-nodes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.027709908Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.027755409Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/catch-all.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.027842722Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/exempt.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.027927024Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/global-default.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.028013547Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/leader-election.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.028094299Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/node-high.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.028178702Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/openshift-control-plane-operators.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.028357847Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/system.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.028485221Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-high.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.028572763Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-low.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.028626995Z cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.028637075Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.028692257Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.028770109Z cluster-scoped-resources/imageregistry.operator.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.028777629Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.028828421Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.028908213Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.028956195Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/cluster.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.029002826Z cluster-scoped-resources/migration.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.029014957Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.029054758Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/console-plugin-storage-version-migration.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.029153261Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-controllerconfig-storage-version-migration.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.029319216Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-machineconfigpool-storage-version-migration.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.029370077Z cluster-scoped-resources/oauth.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.029378267Z cluster-scoped-resources/oauth.openshift.io/oauthclients/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.029419559Z cluster-scoped-resources/oauth.openshift.io/oauthclients/console.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.029523332Z cluster-scoped-resources/operator.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.029530422Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.029559533Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.029650046Z cluster-scoped-resources/operator.openshift.io/consoles/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.029693807Z cluster-scoped-resources/operator.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.02980583Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.029832721Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/cluster.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.029908823Z cluster-scoped-resources/operator.openshift.io/dnses/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.029943984Z cluster-scoped-resources/operator.openshift.io/dnses/default.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.029992126Z cluster-scoped-resources/operator.openshift.io/insightsoperators/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.030038857Z cluster-scoped-resources/operator.openshift.io/insightsoperators/cluster.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.030156031Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.030186592Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/cluster.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.030256154Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.030296005Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/cluster.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.030352296Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.030384917Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/cluster.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.030438189Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.03047865Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/cluster.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.030544382Z cluster-scoped-resources/operator.openshift.io/networks/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.030581443Z cluster-scoped-resources/operator.openshift.io/networks/cluster.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.030660166Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.030686657Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/cluster.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.030744958Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.030788369Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/cluster.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.030837811Z cluster-scoped-resources/operator.openshift.io/servicecas/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.030866682Z cluster-scoped-resources/operator.openshift.io/servicecas/cluster.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.030961725Z cluster-scoped-resources/operator.openshift.io/storages/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.031003366Z cluster-scoped-resources/operator.openshift.io/storages/cluster.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.031102429Z cluster-scoped-resources/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.031109899Z cluster-scoped-resources/operators.coreos.com/olmconfigs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.03114289Z cluster-scoped-resources/operators.coreos.com/olmconfigs/cluster.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.031315525Z cluster-scoped-resources/operators.coreos.com/operators/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.031370427Z cluster-scoped-resources/operators.coreos.com/operators/authorino-operator.kuadrant-system.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.03147065Z cluster-scoped-resources/operators.coreos.com/operators/dns-operator.kuadrant-system.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.031573063Z cluster-scoped-resources/operators.coreos.com/operators/leader-worker-set.openshift-lws-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.031738658Z cluster-scoped-resources/operators.coreos.com/operators/limitador-operator.kuadrant-system.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.031841851Z cluster-scoped-resources/operators.coreos.com/operators/openshift-cert-manager-operator.cert-manager-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.031952984Z cluster-scoped-resources/operators.coreos.com/operators/openshift-custom-metrics-autoscaler-operator.openshift-keda.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.032058657Z cluster-scoped-resources/operators.coreos.com/operators/rhcl-operator.kuadrant-system.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.032190391Z cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.032238443Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.032306585Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/aws-ebs-csi-driver-operator-clusterrolebinding.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.032412338Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cloud-network-config-controller.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.03249347Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-node-tuning-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.032876942Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-storage-operator-role.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.033009506Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.033102189Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-runner-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.033194451Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/metrics-daemon-sa-rolebinding.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.033298994Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-admission-controller-webhook.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.033381597Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-ancillary-tools.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.03347028Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-cluster-readers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.033562532Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-group.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.033648585Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-transient.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.033742778Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-whereabouts.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.03382964Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-diagnostics.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.033914623Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-node-identity.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.034002736Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-image-registry-pruner.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.034094338Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-iptables-alerter.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.034181381Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.034361916Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-identity-limited.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.034448789Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-kube-rbac-proxy.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.034543972Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/registry-registry-role.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.034666535Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.034707666Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/aws-ebs-csi-driver-operator-clusterrole.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.03481094Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cloud-network-config-controller.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.034895682Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cluster-node-tuning-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.034985715Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-extensions-reader.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.035133949Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.035221952Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.035335045Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.035417568Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/helm-chartrepos-viewer.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.03550462Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/metrics-daemon-role.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.035592173Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-admission-controller-webhook.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.035677075Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-ancillary-tools.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.035770168Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.035852841Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/net-attach-def-project.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.035939313Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-diagnostics.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.036027086Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-node-identity.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.036113348Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-csi-snapshot-controller-runner.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.036208471Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-iptables-alerter.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.036356846Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-cluster-reader.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.036443308Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.036547421Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-kube-rbac-proxy.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.036633874Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.036735047Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-editor.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.03682172Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-viewer.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.036909692Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/project-helm-chartrepository-editor.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.037004155Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-admin.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.037120658Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-basic-user.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.037281233Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-storage-admin.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.037376866Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-view.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.037465619Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:registry.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.037565642Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/whereabouts-cni.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.037625733Z cluster-scoped-resources/samples.operator.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.037633024Z cluster-scoped-resources/samples.operator.openshift.io/configs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.037683365Z cluster-scoped-resources/samples.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.037745287Z cluster-scoped-resources/snapshot.storage.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.037758508Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.037804939Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/csi-aws-vsc.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.037871801Z cluster-scoped-resources/storage.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.037880951Z cluster-scoped-resources/storage.k8s.io/csidrivers/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.037923582Z cluster-scoped-resources/storage.k8s.io/csidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.037994864Z cluster-scoped-resources/storage.k8s.io/csinodes/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.038044856Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-128-215.ec2.internal.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.038134729Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-131-203.ec2.internal.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.038243442Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-139-134.ec2.internal.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.038288443Z cluster-scoped-resources/storage.k8s.io/storageclasses/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.038325784Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp2-csi.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.038424457Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp3-csi.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.038473979Z cluster-scoped-resources/storage.k8s.io/volumeattachments/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.03852357Z cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-9ab58ddab07f370d616cd8a1a58d8d7b249d1b020ed628d522c4dc9fafc70a96.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.038588652Z host_service_logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.038623053Z host_service_logs/masters/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.038673825Z host_service_logs/masters/NetworkManager_service.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.038744767Z host_service_logs/masters/crio_service.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.038811109Z host_service_logs/masters/kubelet_service.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.038881291Z host_service_logs/masters/machine-config-daemon-firstboot_service.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.039013535Z host_service_logs/masters/machine-config-daemon-host_service.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.039123178Z host_service_logs/masters/openvswitch_service.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.03919527Z host_service_logs/masters/ostree-finalize-staged_service.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.039302743Z host_service_logs/masters/ovs-configuration_service.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.039376766Z host_service_logs/masters/ovs-vswitchd_service.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.039451438Z host_service_logs/masters/ovsdb-server_service.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.039532081Z host_service_logs/masters/rpm-ostreed_service.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.039580972Z ingress_controllers/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.039595052Z ingress_controllers/default/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.039599023Z ingress_controllers/default/router-default-586b57644b-q24lm/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.039657084Z ingress_controllers/default/router-default-586b57644b-q24lm/haproxy.config [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.039793678Z insights-data/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.039801969Z istio/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.03985174Z istio/aggregated-discovery-api.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.039967294Z istio/aggregated-discovery-apis.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.040619003Z istio/event-filter.html [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.040811929Z istio/timestamp [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.040935722Z istio/cluster-scoped-resources/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.040945863Z istio/cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.040953783Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.041017735Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/istio-sidecar-injector-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.041094967Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.041137228Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/istio-validator-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.04120563Z istio/cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.041313094Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.041364745Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.041553291Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.042672594Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.042890161Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.043069606Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.043593731Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.043777817Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.044266052Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.045126927Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.045294122Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.045417596Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepoolimports.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.04555568Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.045697094Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.045827158Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.045953502Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.046060325Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.046173959Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.046363914Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.04654507Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.046805278Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.046991453Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.047420056Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.04756025Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.047695424Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.047851089Z istio/cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.047861389Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.04790323Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.047962292Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.047974782Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.048022264Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.048119347Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.048299972Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.048376404Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.048427366Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.048525349Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.048680843Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.048755436Z istio/namespaces/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.048763116Z istio/namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.048769806Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.048776086Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.048837848Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.048947571Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-2.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.049035404Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.049068175Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/llmisvc-model-pvc-router-manage-2577e794-kserve-route.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.049190358Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/llmisvc-model-pvc-router-manage-59b9d263-kserve-route.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.049372094Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/llmisvc-model-pvc-router-manage-e8706282-kserve-route.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.049485377Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.04957922Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-2.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.049664813Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-3.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.049760586Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-4.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.049877399Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.049887129Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.049968702Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/llmisvc-model-pvc-router-manage-2577e794-inference-pool.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.050102886Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/llmisvc-model-pvc-router-manage-59b9d263-inference-pool.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.050389174Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/llmisvc-model-pvc-router-manage-e8706282-inference-pool.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.050448336Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.050458116Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.050513488Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/llmisvc-model-pvc-router-manage-2577e794-inference-pool.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.050609601Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/llmisvc-model-pvc-router-manage-59b9d263-inference-pool.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.050751995Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/llmisvc-model-pvc-router-manage-e8706282-inference-pool.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.050819767Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.050847188Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.05090604Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-2577e794-kserve-scheduler.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.050999803Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.051155157Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.05124813Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-59b9d263-kserve-scheduler.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.051362763Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-59b9d263-kserve-shadow-svc.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.051452456Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.051548249Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.051641372Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.051734125Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.051799647Z istio/namespaces/openshift-ingress/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.051833648Z istio/namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.0519223Z istio/namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.051966292Z istio/namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.052019033Z istio/namespaces/openshift-ingress/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.052051364Z istio/namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.052140987Z istio/namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.052431875Z istio/namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.052629791Z istio/namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.052686663Z istio/namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.052726134Z istio/namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.052809387Z istio/namespaces/openshift-ingress/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.052852158Z istio/namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.052939041Z istio/namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.052998042Z istio/namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.053044524Z istio/namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.053125446Z istio/namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.053202428Z istio/namespaces/openshift-ingress/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.05324788Z istio/namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.053859898Z istio/namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.053962901Z istio/namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.054174067Z istio/namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.05427406Z istio/namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.054520698Z istio/namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.054646641Z istio/namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.054848407Z istio/namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.05493724Z istio/namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.054969291Z istio/namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.055049913Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.055057124Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.055095115Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.055165097Z istio/namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.055192508Z istio/namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.05528824Z istio/namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.055336012Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.055423364Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.055503687Z istio/namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.055568989Z istio/namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.05560303Z istio/namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.055670062Z istio/namespaces/openshift-ingress/networking.istio.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.055684972Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.055747604Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-auth-openshift-ai-inference.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.055826707Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-openshift-ai-inference.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.055980311Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/openshift-ai-inference-authn-ssl.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.056072374Z istio/namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.056116685Z istio/namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.056195088Z istio/namespaces/openshift-ingress/pods/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.056207948Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zprl6/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.05628286Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zprl6/istiod-openshift-gateway-94bb8fbfd-zprl6.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.056410094Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zprl6/discovery/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.056418114Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zprl6/discovery/discovery/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.056432845Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zprl6/discovery/discovery/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.056481806Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zprl6/discovery/discovery/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.071242607Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zprl6/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.071321979Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zprl6/discovery/discovery/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.07134229Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-8g97n/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.071396301Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-8g97n/openshift-ai-inference-openshift-default-655f7c58b4-8g97n.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.071543206Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-8g97n/istio-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.071552976Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-8g97n/istio-proxy/istio-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.071568186Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-8g97n/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.071592527Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-8g97n/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.072401751Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-8g97n/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.072635748Z istio/namespaces/openshift-ingress/pods/router-default-586b57644b-q24lm/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.0726806Z istio/namespaces/openshift-ingress/pods/router-default-586b57644b-q24lm/router-default-586b57644b-q24lm.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.072785633Z istio/namespaces/openshift-ingress/pods/router-default-586b57644b-q24lm/router/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.072794313Z istio/namespaces/openshift-ingress/pods/router-default-586b57644b-q24lm/router/router/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.072805763Z istio/namespaces/openshift-ingress/pods/router-default-586b57644b-q24lm/router/router/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.072861675Z istio/namespaces/openshift-ingress/pods/router-default-586b57644b-q24lm/router/router/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.072973488Z istio/namespaces/openshift-ingress/pods/router-default-586b57644b-q24lm/router/router/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.07304404Z istio/namespaces/openshift-ingress/pods/router-default-586b57644b-q24lm/router/router/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.073096372Z istio/namespaces/openshift-ingress/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.073138253Z istio/namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.073189535Z istio/namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.073246476Z istio/namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.073304278Z machine_config_ondisk/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.073313148Z machine_config_termination_logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.073318029Z monitoring/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.073326029Z monitoring/alertmanager/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.07336556Z monitoring/alertmanager/status.json [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.073456053Z monitoring/alertmanager/status.stderr [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.073520164Z monitoring/prometheus/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.073558366Z monitoring/prometheus/alertmanagers.json [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.073651559Z monitoring/prometheus/alertmanagers.stderr [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.073727391Z monitoring/prometheus/rules.json [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.074451743Z monitoring/prometheus/rules.stderr [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.074503494Z monitoring/prometheus/prometheus-k8s-0/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.074584086Z monitoring/prometheus/prometheus-k8s-0/active-targets.json [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.075463583Z monitoring/prometheus/prometheus-k8s-0/active-targets.stderr [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.075521254Z monitoring/prometheus/prometheus-k8s-0/status/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.075571276Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.json [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.075657948Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.stderr [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.075726481Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.json [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.075819293Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.stderr [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.075844844Z monitoring/prometheus/status/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.075889865Z monitoring/prometheus/status/config.json [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.076357339Z monitoring/prometheus/status/config.stderr [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.076428682Z monitoring/prometheus/status/flags.json [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.076533125Z monitoring/prometheus/status/flags.stderr [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.07672422Z namespaces/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.07673168Z namespaces/cert-manager-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.076735691Z namespaces/cert-manager-operator/coordination.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.076741301Z namespaces/cert-manager-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.076777322Z namespaces/cert-manager-operator/coordination.k8s.io/leases/cert-manager-operator-lock.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.076840464Z namespaces/cert-manager-operator/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.076854214Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.076904736Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.077254976Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.077473743Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.077634238Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.077844264Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.078075991Z namespaces/cert-manager-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.078126192Z namespaces/cert-manager-operator/operators.coreos.com/installplans/install-2mdbt.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.078297497Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.078316918Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.07840433Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.078443862Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/openshift-cert-manager-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.078505953Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.078577875Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/openshift-cert-manager-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.078651878Z namespaces/cert-manager/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.078663378Z namespaces/cert-manager/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.078671558Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.07872202Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.078873094Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.07906331Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.079201264Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.079430931Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.079650058Z namespaces/default/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.079694599Z namespaces/default/default.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.079770941Z namespaces/default/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.079814492Z namespaces/default/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.079888405Z namespaces/default/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.079932216Z namespaces/default/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.080016938Z namespaces/default/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.080097591Z namespaces/default/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.080176023Z namespaces/default/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.080236905Z namespaces/default/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.080281376Z namespaces/default/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.080344918Z namespaces/default/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.08038886Z namespaces/default/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.080472432Z namespaces/default/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.080538664Z namespaces/default/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.080585406Z namespaces/default/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.080670148Z namespaces/default/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.080761351Z namespaces/default/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.080806282Z namespaces/default/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.080934126Z namespaces/default/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.081021008Z namespaces/default/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.081194824Z namespaces/default/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.081294827Z namespaces/default/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.081375119Z namespaces/default/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.081501883Z namespaces/default/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.081659238Z namespaces/default/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.08172662Z namespaces/default/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.081770031Z namespaces/default/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.081849493Z namespaces/default/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.081892385Z namespaces/default/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.081967967Z namespaces/default/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.082005298Z namespaces/default/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.082096991Z namespaces/default/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.082174203Z namespaces/default/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.082220534Z namespaces/default/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.082285796Z namespaces/default/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.082334118Z namespaces/default/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.082375739Z namespaces/default/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.08242397Z namespaces/default/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.082431421Z namespaces/default/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.082469622Z namespaces/default/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.082618666Z namespaces/default/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.082792561Z namespaces/default/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.082937526Z namespaces/default/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.083134791Z namespaces/default/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.083337038Z namespaces/default/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.083384679Z namespaces/default/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.08342464Z namespaces/default/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.083476232Z namespaces/default/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.083521003Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.083528833Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.083541314Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/leases/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.083582985Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/leases/epp-kserve-ci-e2e-test-scheduler-ha-replicas-test-inference-pool.llm-d.ai.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.083651377Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.083661617Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.083684738Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.083789931Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-2.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.083856293Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.083866744Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.083913495Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine-default.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.084012118Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.08407159Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.084119561Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.084204943Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.084289766Z namespaces/kserve-ci-e2e-test/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.084307246Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.084361538Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.084517043Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.084695798Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.084834722Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.085031598Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.085280586Z namespaces/kserve/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.085327777Z namespaces/kserve/kserve.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.085401799Z namespaces/kserve/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.085445391Z namespaces/kserve/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.085521523Z namespaces/kserve/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.085564104Z namespaces/kserve/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.085648557Z namespaces/kserve/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.085843142Z namespaces/kserve/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.086038748Z namespaces/kserve/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.086075509Z namespaces/kserve/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.086123361Z namespaces/kserve/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.086170452Z namespaces/kserve/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.086205613Z namespaces/kserve/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.086312306Z namespaces/kserve/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.086385559Z namespaces/kserve/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.08642904Z namespaces/kserve/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.086511692Z namespaces/kserve/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.086556634Z namespaces/kserve/coordination.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.086565184Z namespaces/kserve/coordination.k8s.io/leases/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.086588974Z namespaces/kserve/coordination.k8s.io/leases/kserve-controller-manager-leader-lock.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.086682277Z namespaces/kserve/coordination.k8s.io/leases/llminferenceservice-kserve-controller-manager.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.08676935Z namespaces/kserve/coordination.k8s.io/leases/odh-model-controller.opendatahub.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.086865853Z namespaces/kserve/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.086912484Z namespaces/kserve/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.087166072Z namespaces/kserve/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.087293736Z namespaces/kserve/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.087608085Z namespaces/kserve/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.087687207Z namespaces/kserve/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.087946325Z namespaces/kserve/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.088075259Z namespaces/kserve/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.088357938Z namespaces/kserve/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.08844875Z namespaces/kserve/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.088480611Z namespaces/kserve/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.088613575Z namespaces/kserve/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.088658276Z namespaces/kserve/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.088725078Z namespaces/kserve/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.088795771Z namespaces/kserve/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.088885743Z namespaces/kserve/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.088964136Z namespaces/kserve/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.089031798Z namespaces/kserve/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.089069429Z namespaces/kserve/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.089133791Z namespaces/kserve/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.089177592Z namespaces/kserve/monitoring.coreos.com/servicemonitors/model-serving-api-metrics.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.089300235Z namespaces/kserve/monitoring.coreos.com/servicemonitors/odh-model-controller-metrics-monitor.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.089349797Z namespaces/kserve/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.089401479Z namespaces/kserve/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.08945512Z namespaces/kserve/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.0894674Z namespaces/kserve/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.089521522Z namespaces/kserve/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.089679867Z namespaces/kserve/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.089860032Z namespaces/kserve/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.090000047Z namespaces/kserve/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.090192792Z namespaces/kserve/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.09045583Z namespaces/kserve/pods/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.090478451Z namespaces/kserve/pods/kserve-controller-manager-6b69b6d547-w9wkj/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.090493571Z namespaces/kserve/pods/kserve-controller-manager-6b69b6d547-w9wkj/kserve-controller-manager-6b69b6d547-w9wkj.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.090606214Z namespaces/kserve/pods/kserve-controller-manager-6b69b6d547-w9wkj/manager/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.090616575Z namespaces/kserve/pods/kserve-controller-manager-6b69b6d547-w9wkj/manager/manager/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.090621775Z namespaces/kserve/pods/kserve-controller-manager-6b69b6d547-w9wkj/manager/manager/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.090655326Z namespaces/kserve/pods/kserve-controller-manager-6b69b6d547-w9wkj/manager/manager/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.09078515Z namespaces/kserve/pods/kserve-controller-manager-6b69b6d547-w9wkj/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.090853762Z namespaces/kserve/pods/kserve-controller-manager-6b69b6d547-w9wkj/manager/manager/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.090904403Z namespaces/kserve/pods/llmisvc-controller-manager-dc6dd9667-pq4h9/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.090940075Z namespaces/kserve/pods/llmisvc-controller-manager-dc6dd9667-pq4h9/llmisvc-controller-manager-dc6dd9667-pq4h9.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.091035247Z namespaces/kserve/pods/llmisvc-controller-manager-dc6dd9667-pq4h9/manager/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.091043978Z namespaces/kserve/pods/llmisvc-controller-manager-dc6dd9667-pq4h9/manager/manager/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.091055068Z namespaces/kserve/pods/llmisvc-controller-manager-dc6dd9667-pq4h9/manager/manager/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.091103529Z namespaces/kserve/pods/llmisvc-controller-manager-dc6dd9667-pq4h9/manager/manager/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.095012956Z namespaces/kserve/pods/llmisvc-controller-manager-dc6dd9667-pq4h9/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.095080588Z namespaces/kserve/pods/llmisvc-controller-manager-dc6dd9667-pq4h9/manager/manager/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.09513524Z namespaces/kserve/pods/model-serving-api-fd65d7d6-c4ptz/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.09515757Z namespaces/kserve/pods/model-serving-api-fd65d7d6-c4ptz/model-serving-api-fd65d7d6-c4ptz.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.095289214Z namespaces/kserve/pods/model-serving-api-fd65d7d6-c4ptz/server/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.095301975Z namespaces/kserve/pods/model-serving-api-fd65d7d6-c4ptz/server/server/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.095310635Z namespaces/kserve/pods/model-serving-api-fd65d7d6-c4ptz/server/server/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.095359036Z namespaces/kserve/pods/model-serving-api-fd65d7d6-c4ptz/server/server/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.095452739Z namespaces/kserve/pods/model-serving-api-fd65d7d6-c4ptz/server/server/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.095528241Z namespaces/kserve/pods/model-serving-api-fd65d7d6-c4ptz/server/server/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.095546252Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-gpt9r/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.095596263Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-gpt9r/odh-model-controller-84b54b7f97-gpt9r.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.095675276Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-gpt9r/manager/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.095689206Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-gpt9r/manager/manager/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.095694826Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-gpt9r/manager/manager/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.095754228Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-gpt9r/manager/manager/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.119701833Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-gpt9r/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.119770255Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-gpt9r/manager/manager/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.119811946Z namespaces/kserve/pods/s3-init-ctct5/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.119854497Z namespaces/kserve/pods/s3-init-ctct5/s3-init-ctct5.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.11995041Z namespaces/kserve/pods/s3-init-ctct5/download-hf-model/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.119959741Z namespaces/kserve/pods/s3-init-ctct5/download-hf-model/download-hf-model/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.119964901Z namespaces/kserve/pods/s3-init-ctct5/download-hf-model/download-hf-model/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.119995571Z namespaces/kserve/pods/s3-init-ctct5/download-hf-model/download-hf-model/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.120080774Z namespaces/kserve/pods/s3-init-ctct5/download-hf-model/download-hf-model/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.120154896Z namespaces/kserve/pods/s3-init-ctct5/download-hf-model/download-hf-model/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.120237559Z namespaces/kserve/pods/s3-init-ctct5/s3-init/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.120251529Z namespaces/kserve/pods/s3-init-ctct5/s3-init/s3-init/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.120262199Z namespaces/kserve/pods/s3-init-ctct5/s3-init/s3-init/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.120342412Z namespaces/kserve/pods/s3-init-ctct5/s3-init/s3-init/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.120640151Z namespaces/kserve/pods/s3-init-ctct5/s3-init/s3-init/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.120707073Z namespaces/kserve/pods/s3-init-ctct5/s3-init/s3-init/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.120734173Z namespaces/kserve/pods/seaweedfs-64568bcd49-5tglb/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.120783265Z namespaces/kserve/pods/seaweedfs-64568bcd49-5tglb/seaweedfs-64568bcd49-5tglb.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.120863657Z namespaces/kserve/pods/seaweedfs-64568bcd49-5tglb/seaweedfs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.120875488Z namespaces/kserve/pods/seaweedfs-64568bcd49-5tglb/seaweedfs/seaweedfs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.120884708Z namespaces/kserve/pods/seaweedfs-64568bcd49-5tglb/seaweedfs/seaweedfs/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.12093481Z namespaces/kserve/pods/seaweedfs-64568bcd49-5tglb/seaweedfs/seaweedfs/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.121122025Z namespaces/kserve/pods/seaweedfs-64568bcd49-5tglb/seaweedfs/seaweedfs/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.121190877Z namespaces/kserve/pods/seaweedfs-64568bcd49-5tglb/seaweedfs/seaweedfs/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.121222728Z namespaces/kserve/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.12128308Z namespaces/kserve/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.121353402Z namespaces/kserve/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.121385093Z namespaces/kserve/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.121482616Z namespaces/kuadrant-system/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.121535167Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.121605959Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.12163624Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.121693572Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.121724463Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.121811906Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.122121625Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.122411954Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.122467365Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.122510586Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.122581259Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.12261934Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.122705052Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.122755764Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.122781945Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.122876408Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.122913038Z namespaces/kuadrant-system/coordination.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.122919679Z namespaces/kuadrant-system/coordination.k8s.io/leases/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.12296989Z namespaces/kuadrant-system/coordination.k8s.io/leases/3745a16e.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.123050593Z namespaces/kuadrant-system/coordination.k8s.io/leases/a3f98d6c.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.123135575Z namespaces/kuadrant-system/coordination.k8s.io/leases/aac3a15d.authorino.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.123222108Z namespaces/kuadrant-system/coordination.k8s.io/leases/f139389e.kuadrant.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.12329476Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.123331371Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.123516607Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.12363134Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.124051912Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.124131765Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.124506606Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.12462372Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.124833116Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.124948719Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.124989411Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.125112814Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.125149075Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.125211827Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.125273909Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.125363652Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.125440734Z namespaces/kuadrant-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.125491966Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.125519546Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.125585438Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.125617939Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.125701202Z namespaces/kuadrant-system/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.125708252Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.125758863Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.125961369Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.126268229Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.126429883Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.12664019Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.126844216Z namespaces/kuadrant-system/operators.coreos.com/installplans/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.126874867Z namespaces/kuadrant-system/operators.coreos.com/installplans/install-ds8xx.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.127199626Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.127241918Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.127355141Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.127436773Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.127526196Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.127582928Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.127614719Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/kuadrant.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.127699431Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.127722932Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/authorino-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.127843256Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/dns-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.12799684Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/limitador-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.128094013Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/rhcl-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.128180636Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.128189526Z namespaces/kuadrant-system/pods/authorino-7c77957575-cfthr/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.128237017Z namespaces/kuadrant-system/pods/authorino-7c77957575-cfthr/authorino-7c77957575-cfthr.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.12833438Z namespaces/kuadrant-system/pods/authorino-7c77957575-cfthr/authorino/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.128342051Z namespaces/kuadrant-system/pods/authorino-7c77957575-cfthr/authorino/authorino/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.128348781Z namespaces/kuadrant-system/pods/authorino-7c77957575-cfthr/authorino/authorino/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.128410933Z namespaces/kuadrant-system/pods/authorino-7c77957575-cfthr/authorino/authorino/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.131796834Z namespaces/kuadrant-system/pods/authorino-7c77957575-cfthr/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.131865026Z namespaces/kuadrant-system/pods/authorino-7c77957575-cfthr/authorino/authorino/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.131922067Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xdn96/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.131968189Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xdn96/authorino-operator-6d85f6564-xdn96.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.132181135Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xdn96/manager/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.132191026Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xdn96/manager/manager/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.132196006Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xdn96/manager/manager/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.132240787Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xdn96/manager/manager/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.132455353Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xdn96/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.132521625Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xdn96/manager/manager/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.132573457Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-hphfv/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.132621198Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-hphfv/dns-operator-controller-manager-65b49595d7-hphfv.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.132707691Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-hphfv/manager/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.132716551Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-hphfv/manager/manager/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.132724291Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-hphfv/manager/manager/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.132772053Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-hphfv/manager/manager/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.132886466Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-hphfv/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.132958288Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-hphfv/manager/manager/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.132975889Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-4tc92/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.13303346Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-4tc92/kuadrant-console-plugin-85fd4b8b9d-4tc92.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.133110253Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-4tc92/kuadrant-console-plugin/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.133119013Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-4tc92/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.133124363Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-4tc92/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.133136374Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-4tc92/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.133213596Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-4tc92/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.133316489Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-4tc92/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.13333556Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.133378571Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.133485674Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x/manager/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.133493024Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x/manager/manager/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.133497084Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x/manager/manager/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.133532145Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x/manager/manager/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.145564974Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.145631787Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-cf52x/manager/manager/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.145676568Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-hnvp6/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.145733439Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-hnvp6/limitador-limitador-69574b596d-hnvp6.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.145835113Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-hnvp6/limitador/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.145844613Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-hnvp6/limitador/limitador/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.145850203Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-hnvp6/limitador/limitador/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.145876854Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-hnvp6/limitador/limitador/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.145974097Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-hnvp6/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.146039209Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-hnvp6/limitador/limitador/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.14608799Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-mw8p8/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.146134571Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-mw8p8/limitador-operator-controller-manager-6f9f468797-mw8p8.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.146267715Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-mw8p8/manager/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.146281746Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-mw8p8/manager/manager/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.146286526Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-mw8p8/manager/manager/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.146329487Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-mw8p8/manager/manager/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.146500182Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-mw8p8/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.146570904Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-mw8p8/manager/manager/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.146615846Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.146653107Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.146710999Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.14675415Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.146805722Z namespaces/kube-node-lease/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.146812902Z namespaces/kube-node-lease/coordination.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.146822022Z namespaces/kube-node-lease/coordination.k8s.io/leases/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.146877014Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-128-215.ec2.internal.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.146969956Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-131-203.ec2.internal.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.147055289Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-139-134.ec2.internal.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.147106791Z namespaces/kube-node-lease/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.147120691Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.147175142Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.147374148Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.147585435Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.14774037Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.147955546Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.148197833Z namespaces/kube-public/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.148205223Z namespaces/kube-public/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.148215824Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.148281086Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.14844646Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.148624546Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.14876649Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.148957966Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.149313597Z namespaces/kube-system/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.149353938Z namespaces/kube-system/kube-system.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.14942226Z namespaces/kube-system/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.149457681Z namespaces/kube-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.149512882Z namespaces/kube-system/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.149547103Z namespaces/kube-system/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.149675627Z namespaces/kube-system/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.14975421Z namespaces/kube-system/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.149833952Z namespaces/kube-system/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.149886243Z namespaces/kube-system/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.149917384Z namespaces/kube-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.149951796Z namespaces/kube-system/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.149992587Z namespaces/kube-system/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.150081159Z namespaces/kube-system/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.150124731Z namespaces/kube-system/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.150168412Z namespaces/kube-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.150275275Z namespaces/kube-system/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.150314796Z namespaces/kube-system/coordination.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.150330947Z namespaces/kube-system/coordination.k8s.io/leases/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.150386409Z namespaces/kube-system/coordination.k8s.io/leases/apiserver-kol725mcahbz2es6lrabqjsrm4.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.150473931Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-cainjector-leader-election.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.150562124Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-controller.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.150642526Z namespaces/kube-system/coordination.k8s.io/leases/kube-controller-manager.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.15078341Z namespaces/kube-system/coordination.k8s.io/leases/kube-scheduler.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.150858592Z namespaces/kube-system/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.150904874Z namespaces/kube-system/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.151055128Z namespaces/kube-system/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.151138971Z namespaces/kube-system/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.1514287Z namespaces/kube-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.151505792Z namespaces/kube-system/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.15178873Z namespaces/kube-system/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.151917884Z namespaces/kube-system/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.152260484Z namespaces/kube-system/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.152331626Z namespaces/kube-system/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.152381148Z namespaces/kube-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.152435959Z namespaces/kube-system/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.1524679Z namespaces/kube-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.152522662Z namespaces/kube-system/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.152573894Z namespaces/kube-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.152648016Z namespaces/kube-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.152728528Z namespaces/kube-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.152761779Z namespaces/kube-system/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.15279951Z namespaces/kube-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.152860262Z namespaces/kube-system/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.152885833Z namespaces/kube-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.152941285Z namespaces/kube-system/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.152952485Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.153005656Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.153167541Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.153389418Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.153525312Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.153715338Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.154017997Z namespaces/kube-system/pods/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.15412571Z namespaces/kube-system/pods/global-pull-secret-syncer-66pbz/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.154173731Z namespaces/kube-system/pods/global-pull-secret-syncer-66pbz/global-pull-secret-syncer-66pbz.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.154275374Z namespaces/kube-system/pods/global-pull-secret-syncer-66pbz/global-pull-secret-syncer/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.154287205Z namespaces/kube-system/pods/global-pull-secret-syncer-66pbz/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.154292115Z namespaces/kube-system/pods/global-pull-secret-syncer-66pbz/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.154322236Z namespaces/kube-system/pods/global-pull-secret-syncer-66pbz/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.154669846Z namespaces/kube-system/pods/global-pull-secret-syncer-66pbz/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.154739288Z namespaces/kube-system/pods/global-pull-secret-syncer-66pbz/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.154773149Z namespaces/kube-system/pods/global-pull-secret-syncer-q4wtd/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.15481481Z namespaces/kube-system/pods/global-pull-secret-syncer-q4wtd/global-pull-secret-syncer-q4wtd.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.154890263Z namespaces/kube-system/pods/global-pull-secret-syncer-q4wtd/global-pull-secret-syncer/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.154903823Z namespaces/kube-system/pods/global-pull-secret-syncer-q4wtd/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.154908693Z namespaces/kube-system/pods/global-pull-secret-syncer-q4wtd/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.154927424Z namespaces/kube-system/pods/global-pull-secret-syncer-q4wtd/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.155287655Z namespaces/kube-system/pods/global-pull-secret-syncer-q4wtd/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.155360047Z namespaces/kube-system/pods/global-pull-secret-syncer-q4wtd/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.155395738Z namespaces/kube-system/pods/global-pull-secret-syncer-qlzr5/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.155437019Z namespaces/kube-system/pods/global-pull-secret-syncer-qlzr5/global-pull-secret-syncer-qlzr5.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.155499821Z namespaces/kube-system/pods/global-pull-secret-syncer-qlzr5/global-pull-secret-syncer/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.155510151Z namespaces/kube-system/pods/global-pull-secret-syncer-qlzr5/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.155536592Z namespaces/kube-system/pods/global-pull-secret-syncer-qlzr5/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.155592824Z namespaces/kube-system/pods/global-pull-secret-syncer-qlzr5/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.155948224Z namespaces/kube-system/pods/global-pull-secret-syncer-qlzr5/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.156022197Z namespaces/kube-system/pods/global-pull-secret-syncer-qlzr5/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.156068968Z namespaces/kube-system/pods/konnectivity-agent-4qmlg/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.156115419Z namespaces/kube-system/pods/konnectivity-agent-4qmlg/konnectivity-agent-4qmlg.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.156204222Z namespaces/kube-system/pods/konnectivity-agent-4qmlg/konnectivity-agent/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.156211392Z namespaces/kube-system/pods/konnectivity-agent-4qmlg/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.156220692Z namespaces/kube-system/pods/konnectivity-agent-4qmlg/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.156293815Z namespaces/kube-system/pods/konnectivity-agent-4qmlg/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.157458519Z namespaces/kube-system/pods/konnectivity-agent-4qmlg/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.157523421Z namespaces/kube-system/pods/konnectivity-agent-4qmlg/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.157586993Z namespaces/kube-system/pods/konnectivity-agent-n4lpd/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.157640115Z namespaces/kube-system/pods/konnectivity-agent-n4lpd/konnectivity-agent-n4lpd.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.157725817Z namespaces/kube-system/pods/konnectivity-agent-n4lpd/konnectivity-agent/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.157733397Z namespaces/kube-system/pods/konnectivity-agent-n4lpd/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.157737558Z namespaces/kube-system/pods/konnectivity-agent-n4lpd/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.157769909Z namespaces/kube-system/pods/konnectivity-agent-n4lpd/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.15882462Z namespaces/kube-system/pods/konnectivity-agent-n4lpd/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.158894532Z namespaces/kube-system/pods/konnectivity-agent-n4lpd/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.158924773Z namespaces/kube-system/pods/konnectivity-agent-z9pwv/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.158972165Z namespaces/kube-system/pods/konnectivity-agent-z9pwv/konnectivity-agent-z9pwv.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.159058997Z namespaces/kube-system/pods/konnectivity-agent-z9pwv/konnectivity-agent/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.159067617Z namespaces/kube-system/pods/konnectivity-agent-z9pwv/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.159072448Z namespaces/kube-system/pods/konnectivity-agent-z9pwv/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.159120519Z namespaces/kube-system/pods/konnectivity-agent-z9pwv/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.160410257Z namespaces/kube-system/pods/konnectivity-agent-z9pwv/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.16050099Z namespaces/kube-system/pods/konnectivity-agent-z9pwv/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.160562722Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.160603033Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.161138419Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal/haproxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.16115268Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.16115793Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.16116349Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.16116921Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.161192801Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.161206001Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-203.ec2.internal/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.161268243Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-203.ec2.internal/kube-apiserver-proxy-ip-10-0-131-203.ec2.internal.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.161351996Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-203.ec2.internal/haproxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.161360986Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-203.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.161368106Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-203.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.161423278Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-203.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.161624834Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-203.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.161696546Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-203.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.161716747Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-139-134.ec2.internal/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.161773338Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-139-134.ec2.internal/kube-apiserver-proxy-ip-10-0-139-134.ec2.internal.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.16183714Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-139-134.ec2.internal/haproxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.16184455Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-139-134.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.161849671Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-139-134.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.161883021Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-139-134.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.162282144Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-139-134.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.162355766Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-139-134.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.162383176Z namespaces/kube-system/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.162430978Z namespaces/kube-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.162647404Z namespaces/kube-system/rbac.authorization.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.162658795Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.162703736Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-authentication-reader.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.162801739Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.162862341Z namespaces/kube-system/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.162913022Z namespaces/kube-system/rbac.authorization.k8s.io/roles/extension-apiserver-authentication-reader.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.162964034Z namespaces/kube-system/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.163000635Z namespaces/kube-system/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.163052866Z namespaces/open-cluster-management-1334158e-23ec-4349-bd08-e85981d41/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.163060256Z namespaces/open-cluster-management-1334158e-23ec-4349-bd08-e85981d41/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.163070167Z namespaces/open-cluster-management-1334158e-23ec-4349-bd08-e85981d41/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.163125969Z namespaces/open-cluster-management-1334158e-23ec-4349-bd08-e85981d41/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.163329944Z namespaces/open-cluster-management-1334158e-23ec-4349-bd08-e85981d41/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.16351379Z namespaces/open-cluster-management-1334158e-23ec-4349-bd08-e85981d41/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.163652604Z namespaces/open-cluster-management-1334158e-23ec-4349-bd08-e85981d41/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.163874311Z namespaces/open-cluster-management-1334158e-23ec-4349-bd08-e85981d41/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.164101818Z namespaces/open-cluster-management-agent-addon/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.164113368Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.164117478Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.164155569Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/cluster-proxy.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.164251312Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/work-manager.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.164312074Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.164343815Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.164393876Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.164549671Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.164729486Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.164880851Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.165087597Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.165337815Z namespaces/openshift-apiserver-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.165384896Z namespaces/openshift-apiserver-operator/openshift-apiserver-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.165456918Z namespaces/openshift-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.165498599Z namespaces/openshift-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.165571651Z namespaces/openshift-apiserver-operator/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.165602792Z namespaces/openshift-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.165691435Z namespaces/openshift-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.165770297Z namespaces/openshift-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.16584926Z namespaces/openshift-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.165896381Z namespaces/openshift-apiserver-operator/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.165937933Z namespaces/openshift-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.165985214Z namespaces/openshift-apiserver-operator/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.166027055Z namespaces/openshift-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.166105118Z namespaces/openshift-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.166154399Z namespaces/openshift-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.16620078Z namespaces/openshift-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.166301403Z namespaces/openshift-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.166353225Z namespaces/openshift-apiserver-operator/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.166405256Z namespaces/openshift-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.167440597Z namespaces/openshift-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.167561101Z namespaces/openshift-apiserver-operator/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.167642373Z namespaces/openshift-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.167728056Z namespaces/openshift-apiserver-operator/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.167809108Z namespaces/openshift-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.167939202Z namespaces/openshift-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.168126108Z namespaces/openshift-apiserver-operator/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.16819782Z namespaces/openshift-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.168269102Z namespaces/openshift-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.168334904Z namespaces/openshift-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.168396836Z namespaces/openshift-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.168441137Z namespaces/openshift-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.168486649Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.168569201Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.168651063Z namespaces/openshift-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.168714485Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.168755977Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.168847359Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.168893871Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/openshift-apiserver-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.168952052Z namespaces/openshift-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.168992633Z namespaces/openshift-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.169047105Z namespaces/openshift-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.169054285Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.169086606Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.169257121Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.169447097Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.169597822Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.169806518Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.169997723Z namespaces/openshift-apiserver-operator/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.170035735Z namespaces/openshift-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.170096726Z namespaces/openshift-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.170113687Z namespaces/openshift-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.170245691Z namespaces/openshift-apiserver/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.170285652Z namespaces/openshift-apiserver/openshift-apiserver.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.170358104Z namespaces/openshift-apiserver/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.170399426Z namespaces/openshift-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.170452597Z namespaces/openshift-apiserver/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.170491768Z namespaces/openshift-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.170577931Z namespaces/openshift-apiserver/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.170652793Z namespaces/openshift-apiserver/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.170731496Z namespaces/openshift-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.170782817Z namespaces/openshift-apiserver/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.170817708Z namespaces/openshift-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.17087744Z namespaces/openshift-apiserver/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.170918161Z namespaces/openshift-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.171004254Z namespaces/openshift-apiserver/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.171059175Z namespaces/openshift-apiserver/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.171101107Z namespaces/openshift-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.171182249Z namespaces/openshift-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.171287872Z namespaces/openshift-apiserver/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.171333594Z namespaces/openshift-apiserver/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.171432926Z namespaces/openshift-apiserver/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.171514019Z namespaces/openshift-apiserver/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.171592771Z namespaces/openshift-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.171674584Z namespaces/openshift-apiserver/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.171752166Z namespaces/openshift-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.17188007Z namespaces/openshift-apiserver/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.172030984Z namespaces/openshift-apiserver/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.172480388Z namespaces/openshift-apiserver/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.172532169Z namespaces/openshift-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.172605381Z namespaces/openshift-apiserver/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.172632962Z namespaces/openshift-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.172710144Z namespaces/openshift-apiserver/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.172731825Z namespaces/openshift-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.172835878Z namespaces/openshift-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.172909181Z namespaces/openshift-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.172974163Z namespaces/openshift-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.173025614Z namespaces/openshift-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.173075046Z namespaces/openshift-apiserver/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.173126357Z namespaces/openshift-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.173167948Z namespaces/openshift-apiserver/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.173183339Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.17323737Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.173417716Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.173731205Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.173870109Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.174075675Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.174314523Z namespaces/openshift-apiserver/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.174362344Z namespaces/openshift-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.174440326Z namespaces/openshift-apiserver/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.174478407Z namespaces/openshift-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.174535089Z namespaces/openshift-authentication-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.174542119Z namespaces/openshift-authentication-operator/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.174553879Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.174600271Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/authentication-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.174678343Z namespaces/openshift-authentication-operator/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.174693724Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.174747865Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.174928691Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.175107786Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.17524919Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.175494247Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.175679563Z namespaces/openshift-authentication/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.175688373Z namespaces/openshift-authentication/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.175698584Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.175763486Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.17591446Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.176092685Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.17625035Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.176492737Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.176681613Z namespaces/openshift-cloud-controller-manager/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.176691813Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.176696834Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.176744675Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/cloud-controller-manager.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.176816937Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.176834067Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.176894609Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.177189428Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.177410335Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.177557869Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.177748055Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.177948841Z namespaces/openshift-cloud-credential-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.177956121Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.177960211Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.178021673Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/cloud-credential-operator-leader.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.178098065Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.178107956Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.178159827Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/cloud-credential-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.178239629Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.17825467Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.178378024Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.178536429Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.178717754Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.178854558Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.179050964Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.179311321Z namespaces/openshift-cloud-network-config-controller/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.179354803Z namespaces/openshift-cloud-network-config-controller/openshift-cloud-network-config-controller.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.179436635Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.179467466Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.179541888Z namespaces/openshift-cloud-network-config-controller/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.17958321Z namespaces/openshift-cloud-network-config-controller/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.179678563Z namespaces/openshift-cloud-network-config-controller/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.179751405Z namespaces/openshift-cloud-network-config-controller/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.179834287Z namespaces/openshift-cloud-network-config-controller/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.179882199Z namespaces/openshift-cloud-network-config-controller/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.17992158Z namespaces/openshift-cloud-network-config-controller/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.179973561Z namespaces/openshift-cloud-network-config-controller/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.180015523Z namespaces/openshift-cloud-network-config-controller/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.180094075Z namespaces/openshift-cloud-network-config-controller/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.180132486Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.180181127Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.180289201Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.180363813Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.180374243Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.180420105Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/cloud-network-config-controller-lock.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.180515968Z namespaces/openshift-cloud-network-config-controller/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.180549818Z namespaces/openshift-cloud-network-config-controller/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.180657962Z namespaces/openshift-cloud-network-config-controller/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.180735574Z namespaces/openshift-cloud-network-config-controller/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.180816577Z namespaces/openshift-cloud-network-config-controller/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.180898659Z namespaces/openshift-cloud-network-config-controller/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.180977131Z namespaces/openshift-cloud-network-config-controller/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.181122566Z namespaces/openshift-cloud-network-config-controller/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.181248889Z namespaces/openshift-cloud-network-config-controller/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.181314001Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.181365013Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/cloud-network-config-controller.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.181418544Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.181452505Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.181505797Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.181557008Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.18162066Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.181667572Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.181747324Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.181828216Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.181874698Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.181914569Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.181962661Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.182003882Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.182053943Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.182061303Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.182100325Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.182249049Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.182543708Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.182684742Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.182876848Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.183063144Z namespaces/openshift-cloud-network-config-controller/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.183096274Z namespaces/openshift-cloud-network-config-controller/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.183148946Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.183159476Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.183183237Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/cloud-network-config-controller-rb.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.183241999Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.183318791Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/cloud-network-config-controller.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.183375193Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.183402794Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.183506677Z namespaces/openshift-cluster-csi-drivers/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.183555228Z namespaces/openshift-cluster-csi-drivers/openshift-cluster-csi-drivers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.18362253Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.183648351Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.183732413Z namespaces/openshift-cluster-csi-drivers/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.183758864Z namespaces/openshift-cluster-csi-drivers/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.183925409Z namespaces/openshift-cluster-csi-drivers/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.184005162Z namespaces/openshift-cluster-csi-drivers/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.184084424Z namespaces/openshift-cluster-csi-drivers/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.184136276Z namespaces/openshift-cluster-csi-drivers/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.184162046Z namespaces/openshift-cluster-csi-drivers/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.184245189Z namespaces/openshift-cluster-csi-drivers/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.18430587Z namespaces/openshift-cluster-csi-drivers/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.184405724Z namespaces/openshift-cluster-csi-drivers/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.184466275Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.184510897Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.184590749Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.184648291Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.184655521Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.184691622Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ebs-csi-aws-com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.184785325Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-attacher-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.184873647Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-resizer-ebs-csi-aws-com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.184984531Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-snapshotter-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.185097214Z namespaces/openshift-cluster-csi-drivers/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.185129205Z namespaces/openshift-cluster-csi-drivers/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.185240038Z namespaces/openshift-cluster-csi-drivers/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.185347272Z namespaces/openshift-cluster-csi-drivers/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.185687092Z namespaces/openshift-cluster-csi-drivers/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.185766984Z namespaces/openshift-cluster-csi-drivers/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.18597337Z namespaces/openshift-cluster-csi-drivers/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.186120145Z namespaces/openshift-cluster-csi-drivers/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.186253448Z namespaces/openshift-cluster-csi-drivers/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.186414633Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.186443974Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/aws-ebs-csi-driver-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.186502936Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.186533247Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.186594619Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.18662675Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.186680261Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.186704342Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.186798445Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.186879507Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.186921948Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.18696545Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.187009131Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.187054872Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.187128815Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.187142315Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.187185207Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.187359452Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.187539287Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.187772054Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.18796527Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.188168486Z namespaces/openshift-cluster-csi-drivers/pods/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.188431184Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-b247h/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.188467225Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-b247h/aws-ebs-csi-driver-node-b247h.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.188589508Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-b247h/csi-driver/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.188600539Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-b247h/csi-driver/csi-driver/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.188605569Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-b247h/csi-driver/csi-driver/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.18864395Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-b247h/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.188742333Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-b247h/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.188812045Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-b247h/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.188847126Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-b247h/csi-liveness-probe/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.188857566Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-b247h/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.188867227Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-b247h/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.188912268Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-b247h/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.189002611Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-b247h/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.189064483Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-b247h/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.189112004Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-b247h/csi-node-driver-registrar/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.189130934Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-b247h/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.189137255Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-b247h/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.189175306Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-b247h/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.189295799Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-b247h/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.189367021Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-b247h/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.189411413Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mmtjp/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.189465815Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mmtjp/aws-ebs-csi-driver-node-mmtjp.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.189552947Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mmtjp/csi-driver/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.189562258Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mmtjp/csi-driver/csi-driver/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.189571818Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mmtjp/csi-driver/csi-driver/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.189630389Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mmtjp/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.189726742Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mmtjp/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.189795234Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mmtjp/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.189824085Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mmtjp/csi-liveness-probe/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.189832746Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mmtjp/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.189841056Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mmtjp/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.189895787Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mmtjp/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.18997863Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mmtjp/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.190051852Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mmtjp/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.190073733Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mmtjp/csi-node-driver-registrar/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.190081293Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mmtjp/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.190089753Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mmtjp/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.190138635Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mmtjp/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.190245848Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mmtjp/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.19032382Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mmtjp/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.190377132Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-psj7d/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.190420153Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-psj7d/aws-ebs-csi-driver-node-psj7d.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.190515386Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-psj7d/csi-driver/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.190526366Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-psj7d/csi-driver/csi-driver/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.190534816Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-psj7d/csi-driver/csi-driver/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.190580238Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-psj7d/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.190673271Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-psj7d/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.190740653Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-psj7d/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.190792774Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-psj7d/csi-liveness-probe/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.190803084Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-psj7d/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.190809225Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-psj7d/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.190845456Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-psj7d/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.190935039Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-psj7d/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.19100235Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-psj7d/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.191049332Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-psj7d/csi-node-driver-registrar/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.191056882Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-psj7d/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.191068402Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-psj7d/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.191109584Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-psj7d/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.191202566Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-psj7d/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.191296119Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-psj7d/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.19132697Z namespaces/openshift-cluster-csi-drivers/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.191383212Z namespaces/openshift-cluster-csi-drivers/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.191436373Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.191443623Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.191489435Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/aws-ebs-csi-driver-operator-rolebinding.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.191556177Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.191599158Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/aws-ebs-csi-driver-operator-role.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.19166161Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.191679301Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.191762923Z namespaces/openshift-cluster-machine-approver/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.191773073Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.191781294Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.191816855Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/cluster-machine-approver-leader.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.191872826Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.191891097Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.191930478Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/cluster-machine-approver.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.19198662Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.19199794Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.192047812Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.192204776Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.192418132Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.192556967Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.192751722Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.192952919Z namespaces/openshift-cluster-node-tuning-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.1929865Z namespaces/openshift-cluster-node-tuning-operator/openshift-cluster-node-tuning-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.193078272Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.193108253Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.193193226Z namespaces/openshift-cluster-node-tuning-operator/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.193239357Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.193385272Z namespaces/openshift-cluster-node-tuning-operator/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.193483004Z namespaces/openshift-cluster-node-tuning-operator/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.193560137Z namespaces/openshift-cluster-node-tuning-operator/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.193606188Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.193639799Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/tuned.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.193731392Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.193769563Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.193827595Z namespaces/openshift-cluster-node-tuning-operator/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.193863436Z namespaces/openshift-cluster-node-tuning-operator/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.193944488Z namespaces/openshift-cluster-node-tuning-operator/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.193985419Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.194036571Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.194109143Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.194156024Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.194166315Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.194211606Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/node-tuning-operator-lock.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.194375511Z namespaces/openshift-cluster-node-tuning-operator/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.194396652Z namespaces/openshift-cluster-node-tuning-operator/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.195401622Z namespaces/openshift-cluster-node-tuning-operator/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.195509175Z namespaces/openshift-cluster-node-tuning-operator/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.19566284Z namespaces/openshift-cluster-node-tuning-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.195742282Z namespaces/openshift-cluster-node-tuning-operator/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.195940248Z namespaces/openshift-cluster-node-tuning-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.196087202Z namespaces/openshift-cluster-node-tuning-operator/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.196245217Z namespaces/openshift-cluster-node-tuning-operator/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.19633837Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.196390281Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/cluster-node-tuning-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.196454443Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.196472894Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.196563166Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.196582507Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.196660109Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.1966923Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.196781453Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.196861805Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.196925697Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.196979999Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.197042211Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.197092412Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/node-tuning-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.197162274Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.197212616Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/node-tuning-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.197279838Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.197326539Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.197371231Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.197384901Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.197448273Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.197635618Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.197829914Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.197986089Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.198239346Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.198491954Z namespaces/openshift-cluster-node-tuning-operator/pods/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.198570336Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-bmcpv/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.198653199Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-bmcpv/tuned-bmcpv.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.198766922Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-bmcpv/tuned/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.198782333Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-bmcpv/tuned/tuned/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.198786513Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-bmcpv/tuned/tuned/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.198820314Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-bmcpv/tuned/tuned/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.198938377Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-bmcpv/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.199005119Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-bmcpv/tuned/tuned/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.199067911Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-gwvg9/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.199107712Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-gwvg9/tuned-gwvg9.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.199199355Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-gwvg9/tuned/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.199210235Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-gwvg9/tuned/tuned/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.199215355Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-gwvg9/tuned/tuned/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.199238576Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-gwvg9/tuned/tuned/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.19937834Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-gwvg9/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.199442892Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-gwvg9/tuned/tuned/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.199486254Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-mbnkx/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.199500574Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-mbnkx/tuned-mbnkx.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.19968538Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-mbnkx/tuned/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.19969599Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-mbnkx/tuned/tuned/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.1997018Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-mbnkx/tuned/tuned/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.199727981Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-mbnkx/tuned/tuned/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.199840474Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-mbnkx/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.199910086Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-mbnkx/tuned/tuned/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.199935357Z namespaces/openshift-cluster-node-tuning-operator/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.199985658Z namespaces/openshift-cluster-node-tuning-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.20004326Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.200061911Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.200174834Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.200189364Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.200239036Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-128-215.ec2.internal.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.20037565Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-131-203.ec2.internal.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.200453712Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-139-134.ec2.internal.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.200505324Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.200549805Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/default.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.200641508Z namespaces/openshift-cluster-samples-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.200688559Z namespaces/openshift-cluster-samples-operator/openshift-cluster-samples-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.200769512Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.200785952Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.200864575Z namespaces/openshift-cluster-samples-operator/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.200911846Z namespaces/openshift-cluster-samples-operator/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.200997269Z namespaces/openshift-cluster-samples-operator/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.201109242Z namespaces/openshift-cluster-samples-operator/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.201219445Z namespaces/openshift-cluster-samples-operator/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.201289387Z namespaces/openshift-cluster-samples-operator/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.201334469Z namespaces/openshift-cluster-samples-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.20138314Z namespaces/openshift-cluster-samples-operator/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.201428041Z namespaces/openshift-cluster-samples-operator/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.201513234Z namespaces/openshift-cluster-samples-operator/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.201570966Z namespaces/openshift-cluster-samples-operator/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.201617327Z namespaces/openshift-cluster-samples-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.201697169Z namespaces/openshift-cluster-samples-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.201775612Z namespaces/openshift-cluster-samples-operator/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.201815053Z namespaces/openshift-cluster-samples-operator/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.202001629Z namespaces/openshift-cluster-samples-operator/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.202086761Z namespaces/openshift-cluster-samples-operator/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.202220425Z namespaces/openshift-cluster-samples-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.20238107Z namespaces/openshift-cluster-samples-operator/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.202488923Z namespaces/openshift-cluster-samples-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.202618247Z namespaces/openshift-cluster-samples-operator/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.202794062Z namespaces/openshift-cluster-samples-operator/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.202852204Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.202903885Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.202957867Z namespaces/openshift-cluster-samples-operator/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.203009899Z namespaces/openshift-cluster-samples-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.20307187Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.203119202Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.203196124Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.203297307Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.203348819Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.20339071Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.203443332Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.203485083Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/samples-operator-alerts.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.203577576Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.203607277Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/cluster-samples-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.203675748Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.2037161Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.203770261Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.203780962Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.203789332Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.203969267Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.204154563Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.204312358Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.204527744Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.20471547Z namespaces/openshift-cluster-samples-operator/pods/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.20472617Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-rjnwt/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.204754071Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-rjnwt/cluster-samples-operator-65db858894-rjnwt.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.204859314Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-rjnwt/cluster-samples-operator-watch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.204868304Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-rjnwt/cluster-samples-operator-watch/cluster-samples-operator-watch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.204873374Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-rjnwt/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.204910935Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-rjnwt/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.205006908Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-rjnwt/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.20507247Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-rjnwt/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.205124782Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-rjnwt/cluster-samples-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.205134092Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-rjnwt/cluster-samples-operator/cluster-samples-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.205143922Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-rjnwt/cluster-samples-operator/cluster-samples-operator/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.205184664Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-rjnwt/cluster-samples-operator/cluster-samples-operator/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.205935946Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-rjnwt/cluster-samples-operator/cluster-samples-operator/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.205998358Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-rjnwt/cluster-samples-operator/cluster-samples-operator/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.206046269Z namespaces/openshift-cluster-samples-operator/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.20607751Z namespaces/openshift-cluster-samples-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.206146082Z namespaces/openshift-cluster-samples-operator/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.206190423Z namespaces/openshift-cluster-samples-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.206326108Z namespaces/openshift-cluster-storage-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.206376879Z namespaces/openshift-cluster-storage-operator/openshift-cluster-storage-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.206447571Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.206478292Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.206549354Z namespaces/openshift-cluster-storage-operator/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.206583965Z namespaces/openshift-cluster-storage-operator/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.206678618Z namespaces/openshift-cluster-storage-operator/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.206778791Z namespaces/openshift-cluster-storage-operator/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.206888824Z namespaces/openshift-cluster-storage-operator/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.206943346Z namespaces/openshift-cluster-storage-operator/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.206983657Z namespaces/openshift-cluster-storage-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.207268466Z namespaces/openshift-cluster-storage-operator/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.207317407Z namespaces/openshift-cluster-storage-operator/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.2074017Z namespaces/openshift-cluster-storage-operator/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.207447491Z namespaces/openshift-cluster-storage-operator/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.207490012Z namespaces/openshift-cluster-storage-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.207569065Z namespaces/openshift-cluster-storage-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.207617176Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.207628206Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.207674538Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/data-source-validator-leader.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.20776345Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/snapshot-controller-leader.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.207845033Z namespaces/openshift-cluster-storage-operator/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.207886434Z namespaces/openshift-cluster-storage-operator/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.208007258Z namespaces/openshift-cluster-storage-operator/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.20809205Z namespaces/openshift-cluster-storage-operator/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.208518603Z namespaces/openshift-cluster-storage-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.208626136Z namespaces/openshift-cluster-storage-operator/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.208839353Z namespaces/openshift-cluster-storage-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.209025008Z namespaces/openshift-cluster-storage-operator/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.209199344Z namespaces/openshift-cluster-storage-operator/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.209307777Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.209352998Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/csi-snapshot-controller-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.209447531Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.209477082Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.209556354Z namespaces/openshift-cluster-storage-operator/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.209587165Z namespaces/openshift-cluster-storage-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.209665457Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.209707968Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.209797721Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.209876884Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.209936615Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.209965966Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.210038518Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.21008996Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/prometheus.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.210173842Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.210220714Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/cluster-storage-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.210386739Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.21042804Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.210519743Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.210535973Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.210576845Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.210730099Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.210913024Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.211055239Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.211274185Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.211563314Z namespaces/openshift-cluster-storage-operator/pods/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.211574274Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-vds4p/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.211607665Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-vds4p/volume-data-source-validator-78495597f6-vds4p.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.211688658Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-vds4p/volume-data-source-validator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.211696148Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-vds4p/volume-data-source-validator/volume-data-source-validator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.211700308Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-vds4p/volume-data-source-validator/volume-data-source-validator/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.211731439Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-vds4p/volume-data-source-validator/volume-data-source-validator/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.211832782Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-vds4p/volume-data-source-validator/volume-data-source-validator/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.211901864Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-vds4p/volume-data-source-validator/volume-data-source-validator/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.211936445Z namespaces/openshift-cluster-storage-operator/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.211968386Z namespaces/openshift-cluster-storage-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.212020757Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.212027948Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.212066209Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.212132471Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.212168212Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.212238134Z namespaces/openshift-cluster-storage-operator/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.212301396Z namespaces/openshift-cluster-storage-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.212403089Z namespaces/openshift-cluster-version/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.21242467Z namespaces/openshift-cluster-version/openshift-cluster-version.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.212515242Z namespaces/openshift-cluster-version/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.212534793Z namespaces/openshift-cluster-version/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.212592795Z namespaces/openshift-cluster-version/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.212612175Z namespaces/openshift-cluster-version/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.212700218Z namespaces/openshift-cluster-version/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.21277665Z namespaces/openshift-cluster-version/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.212855322Z namespaces/openshift-cluster-version/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.212907904Z namespaces/openshift-cluster-version/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.212949555Z namespaces/openshift-cluster-version/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.213009527Z namespaces/openshift-cluster-version/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.213055959Z namespaces/openshift-cluster-version/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.213131791Z namespaces/openshift-cluster-version/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.213190612Z namespaces/openshift-cluster-version/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.213239794Z namespaces/openshift-cluster-version/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.213342377Z namespaces/openshift-cluster-version/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.213395709Z namespaces/openshift-cluster-version/coordination.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.213402759Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.21344651Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/version.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.213538383Z namespaces/openshift-cluster-version/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.213566044Z namespaces/openshift-cluster-version/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.213676707Z namespaces/openshift-cluster-version/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.213756089Z namespaces/openshift-cluster-version/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.213853022Z namespaces/openshift-cluster-version/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.213931565Z namespaces/openshift-cluster-version/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.214015147Z namespaces/openshift-cluster-version/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.214145081Z namespaces/openshift-cluster-version/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.214338277Z namespaces/openshift-cluster-version/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.214408149Z namespaces/openshift-cluster-version/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.21445693Z namespaces/openshift-cluster-version/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.214513912Z namespaces/openshift-cluster-version/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.214562013Z namespaces/openshift-cluster-version/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.214608265Z namespaces/openshift-cluster-version/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.214654346Z namespaces/openshift-cluster-version/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.214736649Z namespaces/openshift-cluster-version/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.214813701Z namespaces/openshift-cluster-version/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.214856512Z namespaces/openshift-cluster-version/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.214896743Z namespaces/openshift-cluster-version/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.214953735Z namespaces/openshift-cluster-version/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.214999597Z namespaces/openshift-cluster-version/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.215048368Z namespaces/openshift-cluster-version/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.215061888Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.21511169Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.215282765Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.21546143Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.215617025Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.21580571Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.215998246Z namespaces/openshift-cluster-version/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.216032617Z namespaces/openshift-cluster-version/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.216103449Z namespaces/openshift-cluster-version/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.21613398Z namespaces/openshift-cluster-version/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.216253624Z namespaces/openshift-config-managed/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.216297545Z namespaces/openshift-config-managed/openshift-config-managed.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.216386768Z namespaces/openshift-config-managed/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.216417229Z namespaces/openshift-config-managed/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.216495191Z namespaces/openshift-config-managed/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.216536672Z namespaces/openshift-config-managed/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.216623355Z namespaces/openshift-config-managed/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.216699747Z namespaces/openshift-config-managed/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.21679012Z namespaces/openshift-config-managed/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.216948404Z namespaces/openshift-config-managed/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.216991286Z namespaces/openshift-config-managed/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.217057868Z namespaces/openshift-config-managed/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.217103689Z namespaces/openshift-config-managed/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.217204592Z namespaces/openshift-config-managed/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.217296245Z namespaces/openshift-config-managed/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.217339116Z namespaces/openshift-config-managed/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.217439709Z namespaces/openshift-config-managed/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.217529222Z namespaces/openshift-config-managed/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.217570943Z namespaces/openshift-config-managed/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.221078868Z namespaces/openshift-config-managed/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.22115611Z namespaces/openshift-config-managed/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.221267984Z namespaces/openshift-config-managed/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.221428428Z namespaces/openshift-config-managed/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.22149473Z namespaces/openshift-config-managed/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.221647345Z namespaces/openshift-config-managed/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.221760138Z namespaces/openshift-config-managed/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.22182317Z namespaces/openshift-config-managed/core/configmaps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.221859951Z namespaces/openshift-config-managed/core/configmaps/console-public.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.221957464Z namespaces/openshift-config-managed/core/configmaps/openshift-network-features.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.222003255Z namespaces/openshift-config-managed/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.222051867Z namespaces/openshift-config-managed/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.222089198Z namespaces/openshift-config-managed/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.222136279Z namespaces/openshift-config-managed/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.222177761Z namespaces/openshift-config-managed/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.222262983Z namespaces/openshift-config-managed/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.222331485Z namespaces/openshift-config-managed/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.222415538Z namespaces/openshift-config-managed/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.222457499Z namespaces/openshift-config-managed/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.22250341Z namespaces/openshift-config-managed/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.222562522Z namespaces/openshift-config-managed/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.222610273Z namespaces/openshift-config-managed/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.222661165Z namespaces/openshift-config-managed/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.222671835Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.222724647Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.222925803Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.223121889Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.223305484Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.223510891Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.223776328Z namespaces/openshift-config-managed/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.22381971Z namespaces/openshift-config-managed/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.223875201Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.223886322Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.223928423Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/openshift-network-public-role-binding.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.223986665Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.224036816Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/openshift-network-public-role.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.224103898Z namespaces/openshift-config-managed/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.224143299Z namespaces/openshift-config-managed/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.224245292Z namespaces/openshift-config-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.224262863Z namespaces/openshift-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.224269393Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.224331665Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/config-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.224407367Z namespaces/openshift-config-operator/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.224419558Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.224464479Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.224622894Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.224812479Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.224950743Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.225150089Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.225416677Z namespaces/openshift-config/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.225464539Z namespaces/openshift-config/openshift-config.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.225701846Z namespaces/openshift-config/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.225751257Z namespaces/openshift-config/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.22584678Z namespaces/openshift-config/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.225907462Z namespaces/openshift-config/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.226013725Z namespaces/openshift-config/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.226106148Z namespaces/openshift-config/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.226202101Z namespaces/openshift-config/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.226410477Z namespaces/openshift-config/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.226465039Z namespaces/openshift-config/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.226547621Z namespaces/openshift-config/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.226592192Z namespaces/openshift-config/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.226678465Z namespaces/openshift-config/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.226723436Z namespaces/openshift-config/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.226765878Z namespaces/openshift-config/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.22684497Z namespaces/openshift-config/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.226895592Z namespaces/openshift-config/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.226942993Z namespaces/openshift-config/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.227087597Z namespaces/openshift-config/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.22716504Z namespaces/openshift-config/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.227256082Z namespaces/openshift-config/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.227374966Z namespaces/openshift-config/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.227453698Z namespaces/openshift-config/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.227607383Z namespaces/openshift-config/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.227748247Z namespaces/openshift-config/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.227793798Z namespaces/openshift-config/core/secrets/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.22784066Z namespaces/openshift-config/core/secrets/pull-secret.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.227876951Z namespaces/openshift-config/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.227930532Z namespaces/openshift-config/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.227983114Z namespaces/openshift-config/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.228034715Z namespaces/openshift-config/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.228244392Z namespaces/openshift-config/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.228303563Z namespaces/openshift-config/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.228402206Z namespaces/openshift-config/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.228478379Z namespaces/openshift-config/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.22852209Z namespaces/openshift-config/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.228567641Z namespaces/openshift-config/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.228624503Z namespaces/openshift-config/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.228663264Z namespaces/openshift-config/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.228719106Z namespaces/openshift-config/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.228730396Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.228785218Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.228936012Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.229119928Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.229279383Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.229478829Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.229678004Z namespaces/openshift-config/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.229709625Z namespaces/openshift-config/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.229762827Z namespaces/openshift-config/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.229796418Z namespaces/openshift-config/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.229894241Z namespaces/openshift-console-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.229941612Z namespaces/openshift-console-operator/openshift-console-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.230017105Z namespaces/openshift-console-operator/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.230050515Z namespaces/openshift-console-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.230128348Z namespaces/openshift-console-operator/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.230162409Z namespaces/openshift-console-operator/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.230272332Z namespaces/openshift-console-operator/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.230403126Z namespaces/openshift-console-operator/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.230501399Z namespaces/openshift-console-operator/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.23053666Z namespaces/openshift-console-operator/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.230587902Z namespaces/openshift-console-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.230625163Z namespaces/openshift-console-operator/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.230669824Z namespaces/openshift-console-operator/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.230748356Z namespaces/openshift-console-operator/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.230781617Z namespaces/openshift-console-operator/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.230832059Z namespaces/openshift-console-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.230914781Z namespaces/openshift-console-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.230959883Z namespaces/openshift-console-operator/coordination.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.230967093Z namespaces/openshift-console-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.231006654Z namespaces/openshift-console-operator/coordination.k8s.io/leases/console-operator-lock.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.231070006Z namespaces/openshift-console-operator/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.231114427Z namespaces/openshift-console-operator/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.231739416Z namespaces/openshift-console-operator/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.231817248Z namespaces/openshift-console-operator/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.232164279Z namespaces/openshift-console-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.232259912Z namespaces/openshift-console-operator/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.232394856Z namespaces/openshift-console-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.232519059Z namespaces/openshift-console-operator/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.232698945Z namespaces/openshift-console-operator/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.232816758Z namespaces/openshift-console-operator/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.23286333Z namespaces/openshift-console-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.232931462Z namespaces/openshift-console-operator/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.232961702Z namespaces/openshift-console-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.233032135Z namespaces/openshift-console-operator/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.233074306Z namespaces/openshift-console-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.233159159Z namespaces/openshift-console-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.233248161Z namespaces/openshift-console-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.233328473Z namespaces/openshift-console-operator/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.233374975Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.233432476Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.233477108Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/cluster-monitoring-prometheus-rules.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.23353898Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.233571421Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/console-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.233633282Z namespaces/openshift-console-operator/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.233664194Z namespaces/openshift-console-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.233712615Z namespaces/openshift-console-operator/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.233723585Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.233778467Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.233928441Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.234108237Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.234249681Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.23456168Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.234762476Z namespaces/openshift-console-operator/pods/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.234775367Z namespaces/openshift-console-operator/pods/console-operator-7866497494-s92dx/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.234788507Z namespaces/openshift-console-operator/pods/console-operator-7866497494-s92dx/console-operator-7866497494-s92dx.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.234998063Z namespaces/openshift-console-operator/pods/console-operator-7866497494-s92dx/console-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.235011354Z namespaces/openshift-console-operator/pods/console-operator-7866497494-s92dx/console-operator/console-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.235017554Z namespaces/openshift-console-operator/pods/console-operator-7866497494-s92dx/console-operator/console-operator/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.235057125Z namespaces/openshift-console-operator/pods/console-operator-7866497494-s92dx/console-operator/console-operator/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.235582571Z namespaces/openshift-console-operator/pods/console-operator-7866497494-s92dx/console-operator/console-operator/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.235609972Z namespaces/openshift-console-operator/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.235668273Z namespaces/openshift-console-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.235731305Z namespaces/openshift-console-operator/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.235761676Z namespaces/openshift-console-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.235824248Z namespaces/openshift-console-user-settings/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.235832588Z namespaces/openshift-console-user-settings/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.235841348Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.235931361Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.236078866Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.236285232Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.236543999Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.236735595Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.236961752Z namespaces/openshift-console/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.237026904Z namespaces/openshift-console/openshift-console.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.237366174Z namespaces/openshift-console/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.237419355Z namespaces/openshift-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.237516689Z namespaces/openshift-console/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.23756565Z namespaces/openshift-console/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.237669483Z namespaces/openshift-console/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.237867189Z namespaces/openshift-console/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.238184978Z namespaces/openshift-console/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.238217479Z namespaces/openshift-console/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.238309402Z namespaces/openshift-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.238348393Z namespaces/openshift-console/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.238411085Z namespaces/openshift-console/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.238487297Z namespaces/openshift-console/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.238539229Z namespaces/openshift-console/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.23858676Z namespaces/openshift-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.238661483Z namespaces/openshift-console/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.238730285Z namespaces/openshift-console/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.238764286Z namespaces/openshift-console/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.239415275Z namespaces/openshift-console/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.239500398Z namespaces/openshift-console/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.239874119Z namespaces/openshift-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.239953031Z namespaces/openshift-console/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.240142237Z namespaces/openshift-console/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.240291001Z namespaces/openshift-console/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.240525608Z namespaces/openshift-console/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.24058658Z namespaces/openshift-console/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.240640522Z namespaces/openshift-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.240693303Z namespaces/openshift-console/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.240726334Z namespaces/openshift-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.240798256Z namespaces/openshift-console/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.240833087Z namespaces/openshift-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.2409232Z namespaces/openshift-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.241002942Z namespaces/openshift-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.241060164Z namespaces/openshift-console/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.241110916Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.241158557Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.241190668Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/console.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.241277721Z namespaces/openshift-console/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.241321892Z namespaces/openshift-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.241382454Z namespaces/openshift-console/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.241392024Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.241444235Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.24159996Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.241795416Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.241970211Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.242190398Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.242510507Z namespaces/openshift-console/pods/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.242532948Z namespaces/openshift-console/pods/console-8bbbbb749-r68gb/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.24259382Z namespaces/openshift-console/pods/console-8bbbbb749-r68gb/console-8bbbbb749-r68gb.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.242705623Z namespaces/openshift-console/pods/console-8bbbbb749-r68gb/console/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.242712953Z namespaces/openshift-console/pods/console-8bbbbb749-r68gb/console/console/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.242716913Z namespaces/openshift-console/pods/console-8bbbbb749-r68gb/console/console/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.242758595Z namespaces/openshift-console/pods/console-8bbbbb749-r68gb/console/console/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.242867228Z namespaces/openshift-console/pods/console-8bbbbb749-r68gb/console/console/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.24293774Z namespaces/openshift-console/pods/console-8bbbbb749-r68gb/console/console/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.242972841Z namespaces/openshift-console/pods/downloads-68c9cbb497-9tnzg/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.243014113Z namespaces/openshift-console/pods/downloads-68c9cbb497-9tnzg/downloads-68c9cbb497-9tnzg.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.243109865Z namespaces/openshift-console/pods/downloads-68c9cbb497-9tnzg/download-server/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.243119025Z namespaces/openshift-console/pods/downloads-68c9cbb497-9tnzg/download-server/download-server/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.243127676Z namespaces/openshift-console/pods/downloads-68c9cbb497-9tnzg/download-server/download-server/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.243154977Z namespaces/openshift-console/pods/downloads-68c9cbb497-9tnzg/download-server/download-server/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.243548969Z namespaces/openshift-console/pods/downloads-68c9cbb497-9tnzg/download-server/download-server/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.2436182Z namespaces/openshift-console/pods/downloads-68c9cbb497-9tnzg/download-server/download-server/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.243658822Z namespaces/openshift-console/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.243696763Z namespaces/openshift-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.243753415Z namespaces/openshift-console/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.243795426Z namespaces/openshift-console/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.243906229Z namespaces/openshift-controller-manager-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.24395583Z namespaces/openshift-controller-manager-operator/openshift-controller-manager-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.244016482Z namespaces/openshift-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.244063984Z namespaces/openshift-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.244105825Z namespaces/openshift-controller-manager-operator/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.244141666Z namespaces/openshift-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.244247609Z namespaces/openshift-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.244341282Z namespaces/openshift-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.244422744Z namespaces/openshift-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.244463316Z namespaces/openshift-controller-manager-operator/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.244509987Z namespaces/openshift-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.244581059Z namespaces/openshift-controller-manager-operator/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.244628331Z namespaces/openshift-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.244709053Z namespaces/openshift-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.244765275Z namespaces/openshift-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.244814856Z namespaces/openshift-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.244896399Z namespaces/openshift-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.244966131Z namespaces/openshift-controller-manager-operator/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.245012162Z namespaces/openshift-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.245124285Z namespaces/openshift-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.245200428Z namespaces/openshift-controller-manager-operator/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.245296341Z namespaces/openshift-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.245378923Z namespaces/openshift-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.245458886Z namespaces/openshift-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.245583039Z namespaces/openshift-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.245886628Z namespaces/openshift-controller-manager-operator/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.24593778Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.245984881Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.246043293Z namespaces/openshift-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.246091614Z namespaces/openshift-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.246161176Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.246207308Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.246313771Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.246396033Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.246445155Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.246483576Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.246541708Z namespaces/openshift-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.246583049Z namespaces/openshift-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.2466256Z namespaces/openshift-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.246635361Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.246659821Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.246821086Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.247000931Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.247137185Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.247357102Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.247653601Z namespaces/openshift-controller-manager-operator/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.247698802Z namespaces/openshift-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.247749404Z namespaces/openshift-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.247796875Z namespaces/openshift-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.247885088Z namespaces/openshift-controller-manager/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.247938259Z namespaces/openshift-controller-manager/openshift-controller-manager.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.247990811Z namespaces/openshift-controller-manager/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.248033392Z namespaces/openshift-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.248082354Z namespaces/openshift-controller-manager/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.248126805Z namespaces/openshift-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.248209928Z namespaces/openshift-controller-manager/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.248330771Z namespaces/openshift-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.248415994Z namespaces/openshift-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.248505006Z namespaces/openshift-controller-manager/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.248553898Z namespaces/openshift-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.24861236Z namespaces/openshift-controller-manager/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.248658201Z namespaces/openshift-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.248749024Z namespaces/openshift-controller-manager/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.248797345Z namespaces/openshift-controller-manager/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.248843326Z namespaces/openshift-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.248925859Z namespaces/openshift-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.24897758Z namespaces/openshift-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.249026262Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.249055753Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.249133785Z namespaces/openshift-controller-manager/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.249185937Z namespaces/openshift-controller-manager/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.249318061Z namespaces/openshift-controller-manager/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.249396073Z namespaces/openshift-controller-manager/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.249477355Z namespaces/openshift-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.249559258Z namespaces/openshift-controller-manager/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.24963755Z namespaces/openshift-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.249761754Z namespaces/openshift-controller-manager/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.249920579Z namespaces/openshift-controller-manager/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.250012311Z namespaces/openshift-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.250060373Z namespaces/openshift-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.250113604Z namespaces/openshift-controller-manager/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.250155806Z namespaces/openshift-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.250200017Z namespaces/openshift-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.250244058Z namespaces/openshift-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.250416463Z namespaces/openshift-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.250492516Z namespaces/openshift-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.250701772Z namespaces/openshift-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.250747023Z namespaces/openshift-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.250795155Z namespaces/openshift-controller-manager/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.250827556Z namespaces/openshift-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.250880497Z namespaces/openshift-controller-manager/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.250887948Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.250928289Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.251082513Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.251287099Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.251428653Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.25163204Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.251817085Z namespaces/openshift-controller-manager/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.251857916Z namespaces/openshift-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.251905778Z namespaces/openshift-controller-manager/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.251947919Z namespaces/openshift-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.252038752Z namespaces/openshift-dns-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.252084063Z namespaces/openshift-dns-operator/openshift-dns-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.252143835Z namespaces/openshift-dns-operator/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.252183426Z namespaces/openshift-dns-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.252384082Z namespaces/openshift-dns-operator/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.252423143Z namespaces/openshift-dns-operator/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.252512366Z namespaces/openshift-dns-operator/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.252591148Z namespaces/openshift-dns-operator/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.25267139Z namespaces/openshift-dns-operator/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.252728822Z namespaces/openshift-dns-operator/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.252776994Z namespaces/openshift-dns-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.252822135Z namespaces/openshift-dns-operator/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.252869256Z namespaces/openshift-dns-operator/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.252954179Z namespaces/openshift-dns-operator/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.253012991Z namespaces/openshift-dns-operator/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.253059992Z namespaces/openshift-dns-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.253141475Z namespaces/openshift-dns-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.253209537Z namespaces/openshift-dns-operator/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.253276119Z namespaces/openshift-dns-operator/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.253375812Z namespaces/openshift-dns-operator/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.253456734Z namespaces/openshift-dns-operator/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.253551707Z namespaces/openshift-dns-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.253629709Z namespaces/openshift-dns-operator/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.253706392Z namespaces/openshift-dns-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.253830635Z namespaces/openshift-dns-operator/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.254009021Z namespaces/openshift-dns-operator/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.254085093Z namespaces/openshift-dns-operator/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.254129724Z namespaces/openshift-dns-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.254183586Z namespaces/openshift-dns-operator/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.254237717Z namespaces/openshift-dns-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.254292859Z namespaces/openshift-dns-operator/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.25433803Z namespaces/openshift-dns-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.254419773Z namespaces/openshift-dns-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.254501065Z namespaces/openshift-dns-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.254548587Z namespaces/openshift-dns-operator/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.254590748Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.254641219Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.254684651Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/dns.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.254754483Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.254792504Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/dns-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.254850986Z namespaces/openshift-dns-operator/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.254891077Z namespaces/openshift-dns-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.254938708Z namespaces/openshift-dns-operator/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.254945949Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.254981559Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.255135554Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.255349281Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.255490155Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.255692171Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.255884266Z namespaces/openshift-dns-operator/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.255927008Z namespaces/openshift-dns-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.25598884Z namespaces/openshift-dns-operator/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.256029831Z namespaces/openshift-dns-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.256117053Z namespaces/openshift-dns/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.256163365Z namespaces/openshift-dns/openshift-dns.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.256238007Z namespaces/openshift-dns/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.256284738Z namespaces/openshift-dns/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.25699257Z namespaces/openshift-dns/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.257040811Z namespaces/openshift-dns/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.257205456Z namespaces/openshift-dns/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.257310239Z namespaces/openshift-dns/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.257391391Z namespaces/openshift-dns/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.257435023Z namespaces/openshift-dns/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.257482234Z namespaces/openshift-dns/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.257539626Z namespaces/openshift-dns/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.257580377Z namespaces/openshift-dns/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.25767331Z namespaces/openshift-dns/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.257726872Z namespaces/openshift-dns/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.257772273Z namespaces/openshift-dns/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.257854625Z namespaces/openshift-dns/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.257922487Z namespaces/openshift-dns/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.257965448Z namespaces/openshift-dns/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.258070062Z namespaces/openshift-dns/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.258158714Z namespaces/openshift-dns/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.258514795Z namespaces/openshift-dns/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.258587437Z namespaces/openshift-dns/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.258860515Z namespaces/openshift-dns/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.258980549Z namespaces/openshift-dns/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.259156834Z namespaces/openshift-dns/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.259243417Z namespaces/openshift-dns/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.259286638Z namespaces/openshift-dns/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.259457073Z namespaces/openshift-dns/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.259501064Z namespaces/openshift-dns/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.259567976Z namespaces/openshift-dns/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.259610318Z namespaces/openshift-dns/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.2596905Z namespaces/openshift-dns/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.259768512Z namespaces/openshift-dns/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.259836764Z namespaces/openshift-dns/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.259873625Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.259931967Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.259974519Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/dns-default.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.26002903Z namespaces/openshift-dns/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.260070671Z namespaces/openshift-dns/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.260120673Z namespaces/openshift-dns/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.260128963Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.260167204Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.26035564Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.260555336Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.26068954Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.260888286Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.261068941Z namespaces/openshift-dns/pods/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.261087272Z namespaces/openshift-dns/pods/dns-default-7f688/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.261133583Z namespaces/openshift-dns/pods/dns-default-7f688/dns-default-7f688.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.261241256Z namespaces/openshift-dns/pods/dns-default-7f688/dns/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.261252127Z namespaces/openshift-dns/pods/dns-default-7f688/dns/dns/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.261261617Z namespaces/openshift-dns/pods/dns-default-7f688/dns/dns/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.261308788Z namespaces/openshift-dns/pods/dns-default-7f688/dns/dns/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.261407451Z namespaces/openshift-dns/pods/dns-default-7f688/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.261480533Z namespaces/openshift-dns/pods/dns-default-7f688/dns/dns/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.261516014Z namespaces/openshift-dns/pods/dns-default-7f688/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.261526355Z namespaces/openshift-dns/pods/dns-default-7f688/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.261530345Z namespaces/openshift-dns/pods/dns-default-7f688/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.261575146Z namespaces/openshift-dns/pods/dns-default-7f688/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.26167867Z namespaces/openshift-dns/pods/dns-default-7f688/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.261747852Z namespaces/openshift-dns/pods/dns-default-7f688/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.261769712Z namespaces/openshift-dns/pods/dns-default-jw7lc/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.261820124Z namespaces/openshift-dns/pods/dns-default-jw7lc/dns-default-jw7lc.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.261902826Z namespaces/openshift-dns/pods/dns-default-jw7lc/dns/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.261911826Z namespaces/openshift-dns/pods/dns-default-jw7lc/dns/dns/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.261923127Z namespaces/openshift-dns/pods/dns-default-jw7lc/dns/dns/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.261971178Z namespaces/openshift-dns/pods/dns-default-jw7lc/dns/dns/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.262063811Z namespaces/openshift-dns/pods/dns-default-jw7lc/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.262131933Z namespaces/openshift-dns/pods/dns-default-jw7lc/dns/dns/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.262174944Z namespaces/openshift-dns/pods/dns-default-jw7lc/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.262181474Z namespaces/openshift-dns/pods/dns-default-jw7lc/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.262186675Z namespaces/openshift-dns/pods/dns-default-jw7lc/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.262251636Z namespaces/openshift-dns/pods/dns-default-jw7lc/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.26236297Z namespaces/openshift-dns/pods/dns-default-jw7lc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.262408391Z namespaces/openshift-dns/pods/dns-default-jw7lc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.262466203Z namespaces/openshift-dns/pods/dns-default-xzzrg/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.262506094Z namespaces/openshift-dns/pods/dns-default-xzzrg/dns-default-xzzrg.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.262593557Z namespaces/openshift-dns/pods/dns-default-xzzrg/dns/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.262605277Z namespaces/openshift-dns/pods/dns-default-xzzrg/dns/dns/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.262613737Z namespaces/openshift-dns/pods/dns-default-xzzrg/dns/dns/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.262652468Z namespaces/openshift-dns/pods/dns-default-xzzrg/dns/dns/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.262743471Z namespaces/openshift-dns/pods/dns-default-xzzrg/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.262812463Z namespaces/openshift-dns/pods/dns-default-xzzrg/dns/dns/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.262850585Z namespaces/openshift-dns/pods/dns-default-xzzrg/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.262857695Z namespaces/openshift-dns/pods/dns-default-xzzrg/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.262864615Z namespaces/openshift-dns/pods/dns-default-xzzrg/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.262918776Z namespaces/openshift-dns/pods/dns-default-xzzrg/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.263013249Z namespaces/openshift-dns/pods/dns-default-xzzrg/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.263083441Z namespaces/openshift-dns/pods/dns-default-xzzrg/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.263124853Z namespaces/openshift-dns/pods/node-resolver-pd2tc/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.263156703Z namespaces/openshift-dns/pods/node-resolver-pd2tc/node-resolver-pd2tc.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.263269247Z namespaces/openshift-dns/pods/node-resolver-pd2tc/dns-node-resolver/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.263280817Z namespaces/openshift-dns/pods/node-resolver-pd2tc/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.263290398Z namespaces/openshift-dns/pods/node-resolver-pd2tc/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.263330059Z namespaces/openshift-dns/pods/node-resolver-pd2tc/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.263421331Z namespaces/openshift-dns/pods/node-resolver-pd2tc/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.263485933Z namespaces/openshift-dns/pods/node-resolver-pd2tc/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.263523194Z namespaces/openshift-dns/pods/node-resolver-vxf4w/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.263575726Z namespaces/openshift-dns/pods/node-resolver-vxf4w/node-resolver-vxf4w.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.263671059Z namespaces/openshift-dns/pods/node-resolver-vxf4w/dns-node-resolver/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.263679759Z namespaces/openshift-dns/pods/node-resolver-vxf4w/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.263686759Z namespaces/openshift-dns/pods/node-resolver-vxf4w/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.26371746Z namespaces/openshift-dns/pods/node-resolver-vxf4w/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.263808003Z namespaces/openshift-dns/pods/node-resolver-vxf4w/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.263870875Z namespaces/openshift-dns/pods/node-resolver-vxf4w/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.263898836Z namespaces/openshift-dns/pods/node-resolver-zcq5p/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.263941207Z namespaces/openshift-dns/pods/node-resolver-zcq5p/node-resolver-zcq5p.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.264022669Z namespaces/openshift-dns/pods/node-resolver-zcq5p/dns-node-resolver/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.26403044Z namespaces/openshift-dns/pods/node-resolver-zcq5p/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.26403443Z namespaces/openshift-dns/pods/node-resolver-zcq5p/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.26406233Z namespaces/openshift-dns/pods/node-resolver-zcq5p/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.264151283Z namespaces/openshift-dns/pods/node-resolver-zcq5p/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.264216715Z namespaces/openshift-dns/pods/node-resolver-zcq5p/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.264280597Z namespaces/openshift-dns/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.264302048Z namespaces/openshift-dns/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.2643823Z namespaces/openshift-dns/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.264430161Z namespaces/openshift-dns/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.264506114Z namespaces/openshift-etcd/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.264557725Z namespaces/openshift-etcd/openshift-etcd.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.264608697Z namespaces/openshift-etcd/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.264649318Z namespaces/openshift-etcd/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.26471985Z namespaces/openshift-etcd/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.264748471Z namespaces/openshift-etcd/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.264839114Z namespaces/openshift-etcd/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.264916096Z namespaces/openshift-etcd/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.264998129Z namespaces/openshift-etcd/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.265034459Z namespaces/openshift-etcd/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.265082341Z namespaces/openshift-etcd/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.265122262Z namespaces/openshift-etcd/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.265168194Z namespaces/openshift-etcd/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.265267526Z namespaces/openshift-etcd/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.265327068Z namespaces/openshift-etcd/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.26538017Z namespaces/openshift-etcd/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.265454982Z namespaces/openshift-etcd/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.265536045Z namespaces/openshift-etcd/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.265580866Z namespaces/openshift-etcd/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.265687279Z namespaces/openshift-etcd/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.265765891Z namespaces/openshift-etcd/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.265845264Z namespaces/openshift-etcd/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.265924466Z namespaces/openshift-etcd/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.266018569Z namespaces/openshift-etcd/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.266138572Z namespaces/openshift-etcd/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.26637766Z namespaces/openshift-etcd/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.266438441Z namespaces/openshift-etcd/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.266486573Z namespaces/openshift-etcd/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.266533894Z namespaces/openshift-etcd/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.266577065Z namespaces/openshift-etcd/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.266625097Z namespaces/openshift-etcd/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.266666038Z namespaces/openshift-etcd/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.26674537Z namespaces/openshift-etcd/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.266825313Z namespaces/openshift-etcd/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.266887485Z namespaces/openshift-etcd/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.266936326Z namespaces/openshift-etcd/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.266983608Z namespaces/openshift-etcd/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.26704259Z namespaces/openshift-etcd/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.26705393Z namespaces/openshift-etcd/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.267087361Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.267129172Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.267308467Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.267496283Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.267635527Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.267918046Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.268115702Z namespaces/openshift-etcd/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.268154523Z namespaces/openshift-etcd/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.268211024Z namespaces/openshift-etcd/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.268278226Z namespaces/openshift-etcd/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.26839418Z namespaces/openshift-host-network/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.268443291Z namespaces/openshift-host-network/openshift-host-network.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.268517134Z namespaces/openshift-host-network/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.268532894Z namespaces/openshift-host-network/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.268616277Z namespaces/openshift-host-network/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.268653767Z namespaces/openshift-host-network/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.26873939Z namespaces/openshift-host-network/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.268814652Z namespaces/openshift-host-network/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.268899725Z namespaces/openshift-host-network/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.268957167Z namespaces/openshift-host-network/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.268988788Z namespaces/openshift-host-network/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.26906138Z namespaces/openshift-host-network/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.269106091Z namespaces/openshift-host-network/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.269189454Z namespaces/openshift-host-network/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.269241505Z namespaces/openshift-host-network/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.269298387Z namespaces/openshift-host-network/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.2693919Z namespaces/openshift-host-network/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.269465422Z namespaces/openshift-host-network/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.269501713Z namespaces/openshift-host-network/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.269613416Z namespaces/openshift-host-network/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.269684378Z namespaces/openshift-host-network/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.269763681Z namespaces/openshift-host-network/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.269848833Z namespaces/openshift-host-network/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.269927145Z namespaces/openshift-host-network/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.27007233Z namespaces/openshift-host-network/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.270172283Z namespaces/openshift-host-network/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.270236425Z namespaces/openshift-host-network/core/resourcequotas/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.270287256Z namespaces/openshift-host-network/core/resourcequotas/host-network-namespace-quotas.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.270364709Z namespaces/openshift-host-network/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.27039545Z namespaces/openshift-host-network/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.270475052Z namespaces/openshift-host-network/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.270511873Z namespaces/openshift-host-network/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.270591425Z namespaces/openshift-host-network/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.270639087Z namespaces/openshift-host-network/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.270728349Z namespaces/openshift-host-network/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.270810612Z namespaces/openshift-host-network/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.270856123Z namespaces/openshift-host-network/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.270897385Z namespaces/openshift-host-network/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.270950636Z namespaces/openshift-host-network/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.270976407Z namespaces/openshift-host-network/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.271042919Z namespaces/openshift-host-network/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.271061339Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.271109781Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.271289846Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.271476902Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.271614596Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.271812692Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.272024348Z namespaces/openshift-host-network/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.272066699Z namespaces/openshift-host-network/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.272115321Z namespaces/openshift-host-network/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.272159532Z namespaces/openshift-host-network/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.272252015Z namespaces/openshift-image-registry/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.272313977Z namespaces/openshift-image-registry/openshift-image-registry.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.272390759Z namespaces/openshift-image-registry/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.27243321Z namespaces/openshift-image-registry/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.272480552Z namespaces/openshift-image-registry/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.272523593Z namespaces/openshift-image-registry/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.272630486Z namespaces/openshift-image-registry/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.27276311Z namespaces/openshift-image-registry/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.272958126Z namespaces/openshift-image-registry/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.272999037Z namespaces/openshift-image-registry/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.273047699Z namespaces/openshift-image-registry/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.27308969Z namespaces/openshift-image-registry/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.273137741Z namespaces/openshift-image-registry/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.273243484Z namespaces/openshift-image-registry/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.273320587Z namespaces/openshift-image-registry/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.273359718Z namespaces/openshift-image-registry/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.273444441Z namespaces/openshift-image-registry/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.273491792Z namespaces/openshift-image-registry/coordination.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.273498622Z namespaces/openshift-image-registry/coordination.k8s.io/leases/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.273540953Z namespaces/openshift-image-registry/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.273616966Z namespaces/openshift-image-registry/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.273655067Z namespaces/openshift-image-registry/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.274849012Z namespaces/openshift-image-registry/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.274951206Z namespaces/openshift-image-registry/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.275203063Z namespaces/openshift-image-registry/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.275319037Z namespaces/openshift-image-registry/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.275536823Z namespaces/openshift-image-registry/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.275657236Z namespaces/openshift-image-registry/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.275853272Z namespaces/openshift-image-registry/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.275911514Z namespaces/openshift-image-registry/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.275944015Z namespaces/openshift-image-registry/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.276003427Z namespaces/openshift-image-registry/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.276049528Z namespaces/openshift-image-registry/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.276092869Z namespaces/openshift-image-registry/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.276139391Z namespaces/openshift-image-registry/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.276241534Z namespaces/openshift-image-registry/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.276349137Z namespaces/openshift-image-registry/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.276407309Z namespaces/openshift-image-registry/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.276465681Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.276529942Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.276572824Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-operator-alerts.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.276667207Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-rules.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.276754989Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/imagestreams-rules.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.276817951Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.276865823Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/image-registry.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.276916534Z namespaces/openshift-image-registry/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.276968056Z namespaces/openshift-image-registry/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.277003677Z namespaces/openshift-image-registry/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.277020777Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.277081139Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.277267484Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.2774539Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.277608805Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.277819481Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.278026987Z namespaces/openshift-image-registry/pods/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.278034047Z namespaces/openshift-image-registry/pods/image-registry-7d57676984-24sqj/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.278059358Z namespaces/openshift-image-registry/pods/image-registry-7d57676984-24sqj/image-registry-7d57676984-24sqj.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.278175862Z namespaces/openshift-image-registry/pods/image-registry-7d57676984-24sqj/registry/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.278183792Z namespaces/openshift-image-registry/pods/image-registry-7d57676984-24sqj/registry/registry/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.278190112Z namespaces/openshift-image-registry/pods/image-registry-7d57676984-24sqj/registry/registry/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.278240084Z namespaces/openshift-image-registry/pods/image-registry-7d57676984-24sqj/registry/registry/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.280409888Z namespaces/openshift-image-registry/pods/image-registry-7d57676984-24sqj/registry/registry/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.280495701Z namespaces/openshift-image-registry/pods/image-registry-7d57676984-24sqj/registry/registry/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.280524712Z namespaces/openshift-image-registry/pods/node-ca-gdf7f/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.280586434Z namespaces/openshift-image-registry/pods/node-ca-gdf7f/node-ca-gdf7f.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.280757239Z namespaces/openshift-image-registry/pods/node-ca-gdf7f/node-ca/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.280767159Z namespaces/openshift-image-registry/pods/node-ca-gdf7f/node-ca/node-ca/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.280772209Z namespaces/openshift-image-registry/pods/node-ca-gdf7f/node-ca/node-ca/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.280820131Z namespaces/openshift-image-registry/pods/node-ca-gdf7f/node-ca/node-ca/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.281034717Z namespaces/openshift-image-registry/pods/node-ca-gdf7f/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.28112052Z namespaces/openshift-image-registry/pods/node-ca-gdf7f/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.281179251Z namespaces/openshift-image-registry/pods/node-ca-k2vg4/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.281240813Z namespaces/openshift-image-registry/pods/node-ca-k2vg4/node-ca-k2vg4.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.281328406Z namespaces/openshift-image-registry/pods/node-ca-k2vg4/node-ca/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.281337246Z namespaces/openshift-image-registry/pods/node-ca-k2vg4/node-ca/node-ca/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.281350026Z namespaces/openshift-image-registry/pods/node-ca-k2vg4/node-ca/node-ca/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.281407588Z namespaces/openshift-image-registry/pods/node-ca-k2vg4/node-ca/node-ca/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.281587084Z namespaces/openshift-image-registry/pods/node-ca-k2vg4/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.281649905Z namespaces/openshift-image-registry/pods/node-ca-k2vg4/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.281689547Z namespaces/openshift-image-registry/pods/node-ca-ql9cx/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.281715337Z namespaces/openshift-image-registry/pods/node-ca-ql9cx/node-ca-ql9cx.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.28179526Z namespaces/openshift-image-registry/pods/node-ca-ql9cx/node-ca/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.28180741Z namespaces/openshift-image-registry/pods/node-ca-ql9cx/node-ca/node-ca/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.28181617Z namespaces/openshift-image-registry/pods/node-ca-ql9cx/node-ca/node-ca/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.281870312Z namespaces/openshift-image-registry/pods/node-ca-ql9cx/node-ca/node-ca/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.282040507Z namespaces/openshift-image-registry/pods/node-ca-ql9cx/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.282110249Z namespaces/openshift-image-registry/pods/node-ca-ql9cx/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.28214529Z namespaces/openshift-image-registry/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.282174611Z namespaces/openshift-image-registry/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.282242583Z namespaces/openshift-image-registry/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.282380127Z namespaces/openshift-image-registry/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.282418768Z namespaces/openshift-infra/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.282425438Z namespaces/openshift-infra/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.282429318Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.28247938Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.282624784Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.28280364Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.282948794Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.28314621Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.283404108Z namespaces/openshift-ingress-canary/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.283441149Z namespaces/openshift-ingress-canary/openshift-ingress-canary.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.283531062Z namespaces/openshift-ingress-canary/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.283574133Z namespaces/openshift-ingress-canary/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.283645145Z namespaces/openshift-ingress-canary/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.283675606Z namespaces/openshift-ingress-canary/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.28381257Z namespaces/openshift-ingress-canary/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.283887722Z namespaces/openshift-ingress-canary/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.283969115Z namespaces/openshift-ingress-canary/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.284028236Z namespaces/openshift-ingress-canary/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.284059237Z namespaces/openshift-ingress-canary/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.284137589Z namespaces/openshift-ingress-canary/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.284179081Z namespaces/openshift-ingress-canary/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.284293194Z namespaces/openshift-ingress-canary/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.284368547Z namespaces/openshift-ingress-canary/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.284390367Z namespaces/openshift-ingress-canary/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.28448923Z namespaces/openshift-ingress-canary/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.284578253Z namespaces/openshift-ingress-canary/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.284598423Z namespaces/openshift-ingress-canary/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.284718897Z namespaces/openshift-ingress-canary/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.284803929Z namespaces/openshift-ingress-canary/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.284964104Z namespaces/openshift-ingress-canary/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.285044647Z namespaces/openshift-ingress-canary/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.285206752Z namespaces/openshift-ingress-canary/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.285347516Z namespaces/openshift-ingress-canary/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.285510961Z namespaces/openshift-ingress-canary/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.285568172Z namespaces/openshift-ingress-canary/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.285600903Z namespaces/openshift-ingress-canary/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.285662445Z namespaces/openshift-ingress-canary/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.285710056Z namespaces/openshift-ingress-canary/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.285777189Z namespaces/openshift-ingress-canary/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.28582813Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.285905092Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.285983565Z namespaces/openshift-ingress-canary/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.286034186Z namespaces/openshift-ingress-canary/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.286070667Z namespaces/openshift-ingress-canary/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.286118999Z namespaces/openshift-ingress-canary/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.28615904Z namespaces/openshift-ingress-canary/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.286218452Z namespaces/openshift-ingress-canary/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.286248953Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.286301134Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.286452099Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.286631064Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.286764018Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.286971364Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.287240882Z namespaces/openshift-ingress-canary/pods/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.287255273Z namespaces/openshift-ingress-canary/pods/ingress-canary-4rt5g/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.287302824Z namespaces/openshift-ingress-canary/pods/ingress-canary-4rt5g/ingress-canary-4rt5g.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.287385697Z namespaces/openshift-ingress-canary/pods/ingress-canary-4rt5g/serve-healthcheck-canary/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.287403167Z namespaces/openshift-ingress-canary/pods/ingress-canary-4rt5g/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.287410297Z namespaces/openshift-ingress-canary/pods/ingress-canary-4rt5g/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.287445288Z namespaces/openshift-ingress-canary/pods/ingress-canary-4rt5g/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.287544441Z namespaces/openshift-ingress-canary/pods/ingress-canary-4rt5g/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.287616973Z namespaces/openshift-ingress-canary/pods/ingress-canary-4rt5g/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.287668305Z namespaces/openshift-ingress-canary/pods/ingress-canary-9qzhl/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.287711476Z namespaces/openshift-ingress-canary/pods/ingress-canary-9qzhl/ingress-canary-9qzhl.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.287792749Z namespaces/openshift-ingress-canary/pods/ingress-canary-9qzhl/serve-healthcheck-canary/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.287802849Z namespaces/openshift-ingress-canary/pods/ingress-canary-9qzhl/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.287811459Z namespaces/openshift-ingress-canary/pods/ingress-canary-9qzhl/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.28785272Z namespaces/openshift-ingress-canary/pods/ingress-canary-9qzhl/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.287952193Z namespaces/openshift-ingress-canary/pods/ingress-canary-9qzhl/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.288016865Z namespaces/openshift-ingress-canary/pods/ingress-canary-9qzhl/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.288219391Z namespaces/openshift-ingress-canary/pods/ingress-canary-wnkpx/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.288283653Z namespaces/openshift-ingress-canary/pods/ingress-canary-wnkpx/ingress-canary-wnkpx.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.288373106Z namespaces/openshift-ingress-canary/pods/ingress-canary-wnkpx/serve-healthcheck-canary/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.288381726Z namespaces/openshift-ingress-canary/pods/ingress-canary-wnkpx/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.288390416Z namespaces/openshift-ingress-canary/pods/ingress-canary-wnkpx/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.288437478Z namespaces/openshift-ingress-canary/pods/ingress-canary-wnkpx/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.288542231Z namespaces/openshift-ingress-canary/pods/ingress-canary-wnkpx/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.288606643Z namespaces/openshift-ingress-canary/pods/ingress-canary-wnkpx/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.288642494Z namespaces/openshift-ingress-canary/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.288681965Z namespaces/openshift-ingress-canary/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.288747817Z namespaces/openshift-ingress-canary/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.288791559Z namespaces/openshift-ingress-canary/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.288897531Z namespaces/openshift-ingress-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.288947483Z namespaces/openshift-ingress-operator/openshift-ingress-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.289022535Z namespaces/openshift-ingress-operator/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.289064167Z namespaces/openshift-ingress-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.289123748Z namespaces/openshift-ingress-operator/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.28916961Z namespaces/openshift-ingress-operator/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.289270023Z namespaces/openshift-ingress-operator/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.289352835Z namespaces/openshift-ingress-operator/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.289434548Z namespaces/openshift-ingress-operator/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.289488719Z namespaces/openshift-ingress-operator/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.289540371Z namespaces/openshift-ingress-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.289598663Z namespaces/openshift-ingress-operator/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.289646184Z namespaces/openshift-ingress-operator/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.289727056Z namespaces/openshift-ingress-operator/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.289775888Z namespaces/openshift-ingress-operator/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.289817349Z namespaces/openshift-ingress-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.289898632Z namespaces/openshift-ingress-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.289965563Z namespaces/openshift-ingress-operator/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.290005395Z namespaces/openshift-ingress-operator/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.291141409Z namespaces/openshift-ingress-operator/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.291237452Z namespaces/openshift-ingress-operator/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.291365655Z namespaces/openshift-ingress-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.291442218Z namespaces/openshift-ingress-operator/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.29152536Z namespaces/openshift-ingress-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.291658184Z namespaces/openshift-ingress-operator/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.29185879Z namespaces/openshift-ingress-operator/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.291918732Z namespaces/openshift-ingress-operator/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.291962513Z namespaces/openshift-ingress-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.292030155Z namespaces/openshift-ingress-operator/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.292075657Z namespaces/openshift-ingress-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.292136238Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.292143248Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.29219424Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/default-wildcard.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.292283683Z namespaces/openshift-ingress-operator/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.292329664Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.292430347Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.292511039Z namespaces/openshift-ingress-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.292581372Z namespaces/openshift-ingress-operator/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.292656884Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.292714105Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.292759207Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ingress-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.292835509Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.29287757Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ingress-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.292929562Z namespaces/openshift-ingress-operator/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.292982914Z namespaces/openshift-ingress-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.293040245Z namespaces/openshift-ingress-operator/operator.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.293047306Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.293079776Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/default.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.293169329Z namespaces/openshift-ingress-operator/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.293180079Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.293221821Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.293412186Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.293593282Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.293738086Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.293936342Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.294172189Z namespaces/openshift-ingress-operator/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.29421501Z namespaces/openshift-ingress-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.294298063Z namespaces/openshift-ingress-operator/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.294340534Z namespaces/openshift-ingress-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.294441377Z namespaces/openshift-ingress/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.294489598Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.294680914Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.294721285Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.294791847Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.294837939Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.294917121Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.295145468Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.295361444Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.295407596Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.295462847Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.295517829Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.29555812Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.295635613Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.295696205Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.295742926Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.295821898Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.2958803Z namespaces/openshift-ingress/coordination.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.29589762Z namespaces/openshift-ingress/coordination.k8s.io/leases/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.295939692Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-ca-openshift-gateway.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.296024834Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-deployment-openshift-gateway.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.296106047Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-status-leader-openshift-gateway.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.2962057Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.296263571Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.296700244Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.296795417Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.297020094Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.297098266Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.297363514Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.297485728Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.297693324Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.297797757Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.297829028Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.297925731Z namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.297932781Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.297969972Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.298043035Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.298073796Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.298149128Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.298180039Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.298338353Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.298421876Z namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.298471717Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.298504608Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.29857236Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.298617422Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/openshift-ai-inference-metrics.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.298690544Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.298730035Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/router-default.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.298782167Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.298820858Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.29887498Z namespaces/openshift-ingress/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.29888439Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.298929221Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.299085646Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.299288352Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.299429156Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.299647043Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.299968562Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.299977012Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zprl6/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.300006573Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zprl6/istiod-openshift-gateway-94bb8fbfd-zprl6.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.300125017Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zprl6/discovery/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.300135327Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zprl6/discovery/discovery/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.300144977Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zprl6/discovery/discovery/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.300190179Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zprl6/discovery/discovery/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.314779294Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zprl6/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.314842696Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zprl6/discovery/discovery/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.314901538Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-8g97n/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.314934569Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-8g97n/openshift-ai-inference-openshift-default-655f7c58b4-8g97n.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.31532838Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-8g97n/istio-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.315339941Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-8g97n/istio-proxy/istio-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.315344161Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-8g97n/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.315379262Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-8g97n/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.316154355Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-8g97n/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.316421673Z namespaces/openshift-ingress/pods/router-default-586b57644b-q24lm/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.316460614Z namespaces/openshift-ingress/pods/router-default-586b57644b-q24lm/router-default-586b57644b-q24lm.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.316586958Z namespaces/openshift-ingress/pods/router-default-586b57644b-q24lm/router/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.316594568Z namespaces/openshift-ingress/pods/router-default-586b57644b-q24lm/router/router/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.316608198Z namespaces/openshift-ingress/pods/router-default-586b57644b-q24lm/router/router/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.31665544Z namespaces/openshift-ingress/pods/router-default-586b57644b-q24lm/router/router/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.316768203Z namespaces/openshift-ingress/pods/router-default-586b57644b-q24lm/router/router/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.316835325Z namespaces/openshift-ingress/pods/router-default-586b57644b-q24lm/router/router/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.316887317Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.316932938Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.31698588Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.317021481Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.317123584Z namespaces/openshift-insights/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.317172275Z namespaces/openshift-insights/openshift-insights.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.317251578Z namespaces/openshift-insights/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.317309379Z namespaces/openshift-insights/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.317385352Z namespaces/openshift-insights/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.317416513Z namespaces/openshift-insights/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.317542826Z namespaces/openshift-insights/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.31765264Z namespaces/openshift-insights/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.317754453Z namespaces/openshift-insights/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.317800524Z namespaces/openshift-insights/apps/deployments/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.317841965Z namespaces/openshift-insights/apps/deployments/insights-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.317916917Z namespaces/openshift-insights/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.317958959Z namespaces/openshift-insights/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.318021981Z namespaces/openshift-insights/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.318064092Z namespaces/openshift-insights/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.318146554Z namespaces/openshift-insights/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.318202066Z namespaces/openshift-insights/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.318264798Z namespaces/openshift-insights/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.31834987Z namespaces/openshift-insights/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.318428643Z namespaces/openshift-insights/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.318474064Z namespaces/openshift-insights/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.319366091Z namespaces/openshift-insights/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.319456823Z namespaces/openshift-insights/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.319744782Z namespaces/openshift-insights/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.319872026Z namespaces/openshift-insights/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.320253177Z namespaces/openshift-insights/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.320377791Z namespaces/openshift-insights/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.320528295Z namespaces/openshift-insights/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.320597808Z namespaces/openshift-insights/core/configmaps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.320644609Z namespaces/openshift-insights/core/configmaps/service-ca-bundle.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.320707251Z namespaces/openshift-insights/core/serviceaccounts/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.320756572Z namespaces/openshift-insights/core/serviceaccounts/gather.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.320840805Z namespaces/openshift-insights/core/serviceaccounts/operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.320902857Z namespaces/openshift-insights/core/services/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.320950628Z namespaces/openshift-insights/core/services/metrics.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.3210141Z namespaces/openshift-insights/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.321064071Z namespaces/openshift-insights/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.321125403Z namespaces/openshift-insights/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.321169725Z namespaces/openshift-insights/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.321248027Z namespaces/openshift-insights/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.321295998Z namespaces/openshift-insights/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.321382421Z namespaces/openshift-insights/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.321464863Z namespaces/openshift-insights/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.321523155Z namespaces/openshift-insights/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.321570227Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.321631808Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.32167921Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/insights-prometheus-rules.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.321753802Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.321802613Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/insights-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.321863615Z namespaces/openshift-insights/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.321909607Z namespaces/openshift-insights/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.321953168Z namespaces/openshift-insights/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.321963408Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.3220094Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.322216956Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.322461623Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.322605668Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.322815264Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.323063151Z namespaces/openshift-insights/pods/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.323072342Z namespaces/openshift-insights/pods/insights-operator-68d9dd94cf-tbqhd/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.323110193Z namespaces/openshift-insights/pods/insights-operator-68d9dd94cf-tbqhd/insights-operator-68d9dd94cf-tbqhd.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.323211026Z namespaces/openshift-insights/pods/insights-operator-68d9dd94cf-tbqhd/insights-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.323217996Z namespaces/openshift-insights/pods/insights-operator-68d9dd94cf-tbqhd/insights-operator/insights-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.323245307Z namespaces/openshift-insights/pods/insights-operator-68d9dd94cf-tbqhd/insights-operator/insights-operator/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.323312149Z namespaces/openshift-insights/pods/insights-operator-68d9dd94cf-tbqhd/insights-operator/insights-operator/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.32370179Z namespaces/openshift-insights/pods/insights-operator-68d9dd94cf-tbqhd/insights-operator/insights-operator/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.324002139Z namespaces/openshift-insights/pods/insights-runtime-extractor-2h9jt/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.3240441Z namespaces/openshift-insights/pods/insights-runtime-extractor-2h9jt/insights-runtime-extractor-2h9jt.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.324147974Z namespaces/openshift-insights/pods/insights-runtime-extractor-2h9jt/exporter/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.324156764Z namespaces/openshift-insights/pods/insights-runtime-extractor-2h9jt/exporter/exporter/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.324162944Z namespaces/openshift-insights/pods/insights-runtime-extractor-2h9jt/exporter/exporter/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.324200155Z namespaces/openshift-insights/pods/insights-runtime-extractor-2h9jt/exporter/exporter/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.324325519Z namespaces/openshift-insights/pods/insights-runtime-extractor-2h9jt/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.324392401Z namespaces/openshift-insights/pods/insights-runtime-extractor-2h9jt/exporter/exporter/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.324435202Z namespaces/openshift-insights/pods/insights-runtime-extractor-2h9jt/extractor/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.324442242Z namespaces/openshift-insights/pods/insights-runtime-extractor-2h9jt/extractor/extractor/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.324446962Z namespaces/openshift-insights/pods/insights-runtime-extractor-2h9jt/extractor/extractor/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.324455163Z namespaces/openshift-insights/pods/insights-runtime-extractor-2h9jt/extractor/extractor/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.324573466Z namespaces/openshift-insights/pods/insights-runtime-extractor-2h9jt/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.324640828Z namespaces/openshift-insights/pods/insights-runtime-extractor-2h9jt/extractor/extractor/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.324672009Z namespaces/openshift-insights/pods/insights-runtime-extractor-2h9jt/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.324680779Z namespaces/openshift-insights/pods/insights-runtime-extractor-2h9jt/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.324686919Z namespaces/openshift-insights/pods/insights-runtime-extractor-2h9jt/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.324742181Z namespaces/openshift-insights/pods/insights-runtime-extractor-2h9jt/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.324835424Z namespaces/openshift-insights/pods/insights-runtime-extractor-2h9jt/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.324903586Z namespaces/openshift-insights/pods/insights-runtime-extractor-2h9jt/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.324952347Z namespaces/openshift-insights/pods/insights-runtime-extractor-7kmgq/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.324987348Z namespaces/openshift-insights/pods/insights-runtime-extractor-7kmgq/insights-runtime-extractor-7kmgq.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.325081431Z namespaces/openshift-insights/pods/insights-runtime-extractor-7kmgq/exporter/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.325088342Z namespaces/openshift-insights/pods/insights-runtime-extractor-7kmgq/exporter/exporter/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.325097342Z namespaces/openshift-insights/pods/insights-runtime-extractor-7kmgq/exporter/exporter/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.325157944Z namespaces/openshift-insights/pods/insights-runtime-extractor-7kmgq/exporter/exporter/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.325243976Z namespaces/openshift-insights/pods/insights-runtime-extractor-7kmgq/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.325326029Z namespaces/openshift-insights/pods/insights-runtime-extractor-7kmgq/exporter/exporter/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.32535255Z namespaces/openshift-insights/pods/insights-runtime-extractor-7kmgq/extractor/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.32537596Z namespaces/openshift-insights/pods/insights-runtime-extractor-7kmgq/extractor/extractor/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.32538053Z namespaces/openshift-insights/pods/insights-runtime-extractor-7kmgq/extractor/extractor/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.325427142Z namespaces/openshift-insights/pods/insights-runtime-extractor-7kmgq/extractor/extractor/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.325520084Z namespaces/openshift-insights/pods/insights-runtime-extractor-7kmgq/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.325590757Z namespaces/openshift-insights/pods/insights-runtime-extractor-7kmgq/extractor/extractor/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.325630458Z namespaces/openshift-insights/pods/insights-runtime-extractor-7kmgq/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.325753462Z namespaces/openshift-insights/pods/insights-runtime-extractor-7kmgq/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.325763442Z namespaces/openshift-insights/pods/insights-runtime-extractor-7kmgq/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.325792703Z namespaces/openshift-insights/pods/insights-runtime-extractor-7kmgq/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.325957288Z namespaces/openshift-insights/pods/insights-runtime-extractor-7kmgq/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.326022279Z namespaces/openshift-insights/pods/insights-runtime-extractor-7kmgq/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.326071131Z namespaces/openshift-insights/pods/insights-runtime-extractor-xmzjq/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.326100772Z namespaces/openshift-insights/pods/insights-runtime-extractor-xmzjq/insights-runtime-extractor-xmzjq.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.326193764Z namespaces/openshift-insights/pods/insights-runtime-extractor-xmzjq/exporter/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.326203975Z namespaces/openshift-insights/pods/insights-runtime-extractor-xmzjq/exporter/exporter/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.326214485Z namespaces/openshift-insights/pods/insights-runtime-extractor-xmzjq/exporter/exporter/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.326242126Z namespaces/openshift-insights/pods/insights-runtime-extractor-xmzjq/exporter/exporter/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.326340839Z namespaces/openshift-insights/pods/insights-runtime-extractor-xmzjq/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.326415171Z namespaces/openshift-insights/pods/insights-runtime-extractor-xmzjq/exporter/exporter/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.326443432Z namespaces/openshift-insights/pods/insights-runtime-extractor-xmzjq/extractor/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.326449652Z namespaces/openshift-insights/pods/insights-runtime-extractor-xmzjq/extractor/extractor/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.326455702Z namespaces/openshift-insights/pods/insights-runtime-extractor-xmzjq/extractor/extractor/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.326497464Z namespaces/openshift-insights/pods/insights-runtime-extractor-xmzjq/extractor/extractor/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.326593716Z namespaces/openshift-insights/pods/insights-runtime-extractor-xmzjq/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.326662809Z namespaces/openshift-insights/pods/insights-runtime-extractor-xmzjq/extractor/extractor/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.326692119Z namespaces/openshift-insights/pods/insights-runtime-extractor-xmzjq/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.32669835Z namespaces/openshift-insights/pods/insights-runtime-extractor-xmzjq/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.32670445Z namespaces/openshift-insights/pods/insights-runtime-extractor-xmzjq/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.326747221Z namespaces/openshift-insights/pods/insights-runtime-extractor-xmzjq/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.326837954Z namespaces/openshift-insights/pods/insights-runtime-extractor-xmzjq/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.326906406Z namespaces/openshift-insights/pods/insights-runtime-extractor-xmzjq/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.326941317Z namespaces/openshift-insights/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.326969748Z namespaces/openshift-insights/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.327030109Z namespaces/openshift-insights/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.327072521Z namespaces/openshift-insights/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.327127903Z namespaces/openshift-keda/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.327134913Z namespaces/openshift-keda/coordination.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.327138833Z namespaces/openshift-keda/coordination.k8s.io/leases/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.327184274Z namespaces/openshift-keda/coordination.k8s.io/leases/olm-operator.keda.sh.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.327296337Z namespaces/openshift-keda/coordination.k8s.io/leases/operator.keda.sh.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.327349049Z namespaces/openshift-keda/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.327355769Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.327403311Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/keda-olm-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.327451662Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.327496394Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-admission-webhooks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.327580136Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-metrics-apiserver.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.327665399Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.32772486Z namespaces/openshift-keda/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.32773023Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.327770062Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.327918166Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.328095781Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/custom-metrics-autoscaler.v2.19.0-2.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.328307658Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.328525634Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.328679989Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.328848534Z namespaces/openshift-keda/operators.coreos.com/installplans/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.328874375Z namespaces/openshift-keda/operators.coreos.com/installplans/install-ttpqk.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.328999208Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.32904093Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/custom-metrics-autoscaler.v2.19.0-2.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.329090701Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.329138723Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/openshift-keda.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.329190364Z namespaces/openshift-keda/operators.coreos.com/subscriptions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.329206674Z namespaces/openshift-keda/operators.coreos.com/subscriptions/openshift-custom-metrics-autoscaler-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.329350539Z namespaces/openshift-kube-apiserver-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.32937983Z namespaces/openshift-kube-apiserver-operator/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.329449002Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.329495583Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.329551065Z namespaces/openshift-kube-apiserver-operator/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.329578396Z namespaces/openshift-kube-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.329672748Z namespaces/openshift-kube-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.329750011Z namespaces/openshift-kube-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.329832433Z namespaces/openshift-kube-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.329882285Z namespaces/openshift-kube-apiserver-operator/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.329922206Z namespaces/openshift-kube-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.329980088Z namespaces/openshift-kube-apiserver-operator/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.330018359Z namespaces/openshift-kube-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.330098541Z namespaces/openshift-kube-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.330147793Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.330197054Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.330295127Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.330349799Z namespaces/openshift-kube-apiserver-operator/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.33040176Z namespaces/openshift-kube-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.330506733Z namespaces/openshift-kube-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.330586706Z namespaces/openshift-kube-apiserver-operator/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.330666838Z namespaces/openshift-kube-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.33074526Z namespaces/openshift-kube-apiserver-operator/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.330826263Z namespaces/openshift-kube-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.330954226Z namespaces/openshift-kube-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.331125052Z namespaces/openshift-kube-apiserver-operator/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.331160063Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.331211794Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.331281596Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.331325218Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.331375219Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.33141096Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.331496493Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.331572675Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.331624427Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.331664348Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.331714309Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.331758981Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.331822352Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.331831853Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.331889214Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.332040319Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.332219234Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.332530014Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.332728639Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.332911775Z namespaces/openshift-kube-apiserver-operator/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.332945616Z namespaces/openshift-kube-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.332991227Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.333022718Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.333138702Z namespaces/openshift-kube-apiserver/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.333178993Z namespaces/openshift-kube-apiserver/openshift-kube-apiserver.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.333247645Z namespaces/openshift-kube-apiserver/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.333303887Z namespaces/openshift-kube-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.333373779Z namespaces/openshift-kube-apiserver/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.33342252Z namespaces/openshift-kube-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.333502753Z namespaces/openshift-kube-apiserver/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.333583125Z namespaces/openshift-kube-apiserver/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.333661818Z namespaces/openshift-kube-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.333718679Z namespaces/openshift-kube-apiserver/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.333761081Z namespaces/openshift-kube-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.333822672Z namespaces/openshift-kube-apiserver/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.333867554Z namespaces/openshift-kube-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.333947226Z namespaces/openshift-kube-apiserver/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.333994587Z namespaces/openshift-kube-apiserver/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.334040329Z namespaces/openshift-kube-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.334118391Z namespaces/openshift-kube-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.334179813Z namespaces/openshift-kube-apiserver/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.334208534Z namespaces/openshift-kube-apiserver/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.334341648Z namespaces/openshift-kube-apiserver/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.33441911Z namespaces/openshift-kube-apiserver/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.334493122Z namespaces/openshift-kube-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.334575585Z namespaces/openshift-kube-apiserver/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.334655847Z namespaces/openshift-kube-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.33476994Z namespaces/openshift-kube-apiserver/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.334934125Z namespaces/openshift-kube-apiserver/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.334993407Z namespaces/openshift-kube-apiserver/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.335032958Z namespaces/openshift-kube-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.33509095Z namespaces/openshift-kube-apiserver/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.335138471Z namespaces/openshift-kube-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.335195983Z namespaces/openshift-kube-apiserver/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.335242975Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.335343538Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.33541737Z namespaces/openshift-kube-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.335489872Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.335532833Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.335587135Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.335623766Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/api-usage.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.3357444Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/podsecurity.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.335887884Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.335934125Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/openshift-kube-apiserver.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.335995437Z namespaces/openshift-kube-apiserver/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.336037508Z namespaces/openshift-kube-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.336111431Z namespaces/openshift-kube-apiserver/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.336119261Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.336158972Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.336334667Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.336527953Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.336672427Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.336873513Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.337060559Z namespaces/openshift-kube-apiserver/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.337086989Z namespaces/openshift-kube-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.337149731Z namespaces/openshift-kube-apiserver/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.337179592Z namespaces/openshift-kube-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.337253834Z namespaces/openshift-kube-controller-manager-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.337330247Z namespaces/openshift-kube-controller-manager-operator/openshift-kube-controller-manager-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.337395589Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.33742764Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.337494412Z namespaces/openshift-kube-controller-manager-operator/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.337541033Z namespaces/openshift-kube-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.337617165Z namespaces/openshift-kube-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.337713408Z namespaces/openshift-kube-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.33778183Z namespaces/openshift-kube-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.337835422Z namespaces/openshift-kube-controller-manager-operator/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.337868113Z namespaces/openshift-kube-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.337925165Z namespaces/openshift-kube-controller-manager-operator/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.337960106Z namespaces/openshift-kube-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.338051678Z namespaces/openshift-kube-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.33810332Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.338153821Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.338244684Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.338301566Z namespaces/openshift-kube-controller-manager-operator/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.338352327Z namespaces/openshift-kube-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.33845639Z namespaces/openshift-kube-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.338536713Z namespaces/openshift-kube-controller-manager-operator/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.338625886Z namespaces/openshift-kube-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.339200313Z namespaces/openshift-kube-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.339297026Z namespaces/openshift-kube-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.339426299Z namespaces/openshift-kube-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.339602015Z namespaces/openshift-kube-controller-manager-operator/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.339666277Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.339718398Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.339752529Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.33979675Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.339866533Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.339918644Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.339994067Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.340074299Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.340128311Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.340150481Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.340221503Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.340348047Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/kube-controller-manager-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.340405019Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.34044107Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.340489471Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.340505712Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.340545653Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.34079944Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.340984916Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.34112101Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.341347437Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.341544123Z namespaces/openshift-kube-controller-manager-operator/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.341578974Z namespaces/openshift-kube-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.341621025Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.341673526Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.341745339Z namespaces/openshift-kube-controller-manager/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.34178521Z namespaces/openshift-kube-controller-manager/openshift-kube-controller-manager.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.341848682Z namespaces/openshift-kube-controller-manager/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.341878733Z namespaces/openshift-kube-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.341950815Z namespaces/openshift-kube-controller-manager/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.341982796Z namespaces/openshift-kube-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.342073738Z namespaces/openshift-kube-controller-manager/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.342152561Z namespaces/openshift-kube-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.342247223Z namespaces/openshift-kube-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.342315416Z namespaces/openshift-kube-controller-manager/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.342365317Z namespaces/openshift-kube-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.342421969Z namespaces/openshift-kube-controller-manager/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.342461Z namespaces/openshift-kube-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.342549263Z namespaces/openshift-kube-controller-manager/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.342605244Z namespaces/openshift-kube-controller-manager/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.342648195Z namespaces/openshift-kube-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.342736098Z namespaces/openshift-kube-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.34279214Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.34280066Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.342838411Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/cluster-policy-controller-lock.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.342933024Z namespaces/openshift-kube-controller-manager/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.342963035Z namespaces/openshift-kube-controller-manager/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.343074058Z namespaces/openshift-kube-controller-manager/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.34314859Z namespaces/openshift-kube-controller-manager/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.34345717Z namespaces/openshift-kube-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.343536362Z namespaces/openshift-kube-controller-manager/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.343621555Z namespaces/openshift-kube-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.343749148Z namespaces/openshift-kube-controller-manager/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.343899703Z namespaces/openshift-kube-controller-manager/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.343950954Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.343986795Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.344040177Z namespaces/openshift-kube-controller-manager/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.344082798Z namespaces/openshift-kube-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.34413818Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.344181141Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.344286335Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.344389378Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.344444159Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.34447802Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.344545052Z namespaces/openshift-kube-controller-manager/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.344579823Z namespaces/openshift-kube-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.344633335Z namespaces/openshift-kube-controller-manager/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.344646975Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.344702327Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.344849161Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.345023526Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.345162901Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.345389957Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.345652805Z namespaces/openshift-kube-controller-manager/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.345686336Z namespaces/openshift-kube-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.345725887Z namespaces/openshift-kube-controller-manager/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.345778109Z namespaces/openshift-kube-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.345844681Z namespaces/openshift-kube-scheduler-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.345876752Z namespaces/openshift-kube-scheduler-operator/openshift-kube-scheduler-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.345977545Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.346007706Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.346072618Z namespaces/openshift-kube-scheduler-operator/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.346103089Z namespaces/openshift-kube-scheduler-operator/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.346192511Z namespaces/openshift-kube-scheduler-operator/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.346294455Z namespaces/openshift-kube-scheduler-operator/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.346371677Z namespaces/openshift-kube-scheduler-operator/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.346420498Z namespaces/openshift-kube-scheduler-operator/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.34646883Z namespaces/openshift-kube-scheduler-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.346517011Z namespaces/openshift-kube-scheduler-operator/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.346548812Z namespaces/openshift-kube-scheduler-operator/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.346640915Z namespaces/openshift-kube-scheduler-operator/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.346684046Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.346733308Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.34681019Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.346859201Z namespaces/openshift-kube-scheduler-operator/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.346908223Z namespaces/openshift-kube-scheduler-operator/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.347010016Z namespaces/openshift-kube-scheduler-operator/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.347081308Z namespaces/openshift-kube-scheduler-operator/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.34716547Z namespaces/openshift-kube-scheduler-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.347248653Z namespaces/openshift-kube-scheduler-operator/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.347378987Z namespaces/openshift-kube-scheduler-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.34749145Z namespaces/openshift-kube-scheduler-operator/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.347641525Z namespaces/openshift-kube-scheduler-operator/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.347693396Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.347742078Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.347780659Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.34782728Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.347880112Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.347929753Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.348004575Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.348086238Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.348136829Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.34817134Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.348221442Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.348363006Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.348396737Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.348413967Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.348464859Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.348612153Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.348792209Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.348937223Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.349122689Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.349326255Z namespaces/openshift-kube-scheduler-operator/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.349375246Z namespaces/openshift-kube-scheduler-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.349410257Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.349453649Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.349535351Z namespaces/openshift-kube-scheduler/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.349571322Z namespaces/openshift-kube-scheduler/openshift-kube-scheduler.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.349633114Z namespaces/openshift-kube-scheduler/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.349666665Z namespaces/openshift-kube-scheduler/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.349721587Z namespaces/openshift-kube-scheduler/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.349787419Z namespaces/openshift-kube-scheduler/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.349922073Z namespaces/openshift-kube-scheduler/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.350009745Z namespaces/openshift-kube-scheduler/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.350099558Z namespaces/openshift-kube-scheduler/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.350148339Z namespaces/openshift-kube-scheduler/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.350192871Z namespaces/openshift-kube-scheduler/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.350297954Z namespaces/openshift-kube-scheduler/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.350366816Z namespaces/openshift-kube-scheduler/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.350451968Z namespaces/openshift-kube-scheduler/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.35049781Z namespaces/openshift-kube-scheduler/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.350543161Z namespaces/openshift-kube-scheduler/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.350630954Z namespaces/openshift-kube-scheduler/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.350698406Z namespaces/openshift-kube-scheduler/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.350749587Z namespaces/openshift-kube-scheduler/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.35084926Z namespaces/openshift-kube-scheduler/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.350931313Z namespaces/openshift-kube-scheduler/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.351008745Z namespaces/openshift-kube-scheduler/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.351087487Z namespaces/openshift-kube-scheduler/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.35116965Z namespaces/openshift-kube-scheduler/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.351326874Z namespaces/openshift-kube-scheduler/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.35149796Z namespaces/openshift-kube-scheduler/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.351550931Z namespaces/openshift-kube-scheduler/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.351599723Z namespaces/openshift-kube-scheduler/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.351639954Z namespaces/openshift-kube-scheduler/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.351686295Z namespaces/openshift-kube-scheduler/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.351735867Z namespaces/openshift-kube-scheduler/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.351765628Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.35185512Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.351936353Z namespaces/openshift-kube-scheduler/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.351985094Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.352032006Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.352075317Z namespaces/openshift-kube-scheduler/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.352104098Z namespaces/openshift-kube-scheduler/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.35217413Z namespaces/openshift-kube-scheduler/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.35218116Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.352239642Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.352425607Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.352605153Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.352739917Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.353042596Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.353242722Z namespaces/openshift-kube-scheduler/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.353291443Z namespaces/openshift-kube-scheduler/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.353358445Z namespaces/openshift-kube-scheduler/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.353398947Z namespaces/openshift-kube-scheduler/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.353469209Z namespaces/openshift-kube-storage-version-migrator-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.35351095Z namespaces/openshift-kube-storage-version-migrator-operator/openshift-kube-storage-version-migrator-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.353581922Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.353624903Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.353670124Z namespaces/openshift-kube-storage-version-migrator-operator/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.353717446Z namespaces/openshift-kube-storage-version-migrator-operator/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.353800428Z namespaces/openshift-kube-storage-version-migrator-operator/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.353933502Z namespaces/openshift-kube-storage-version-migrator-operator/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.354033465Z namespaces/openshift-kube-storage-version-migrator-operator/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.354108048Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.354155339Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.35420049Z namespaces/openshift-kube-storage-version-migrator-operator/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.354249162Z namespaces/openshift-kube-storage-version-migrator-operator/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.354363045Z namespaces/openshift-kube-storage-version-migrator-operator/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.354412117Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.354442898Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.35453866Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.354595582Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.354605522Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.354662474Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/openshift-kube-storage-version-migrator-operator-lock.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.354734266Z namespaces/openshift-kube-storage-version-migrator-operator/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.354780008Z namespaces/openshift-kube-storage-version-migrator-operator/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.354895011Z namespaces/openshift-kube-storage-version-migrator-operator/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.354979143Z namespaces/openshift-kube-storage-version-migrator-operator/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.355156369Z namespaces/openshift-kube-storage-version-migrator-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.355236541Z namespaces/openshift-kube-storage-version-migrator-operator/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.355403626Z namespaces/openshift-kube-storage-version-migrator-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.35552625Z namespaces/openshift-kube-storage-version-migrator-operator/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.355713396Z namespaces/openshift-kube-storage-version-migrator-operator/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.355776137Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.355808628Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.35587272Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.355924492Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.355974373Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.356008774Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.356095677Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.356175589Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.356240371Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.356409466Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.356473938Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.356518549Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.356565891Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.356573421Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.356622692Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.356776357Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.356960973Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.357186989Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.357428027Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.357635273Z namespaces/openshift-kube-storage-version-migrator-operator/pods/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.357645493Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-799cb7f958-9pt94/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.357672884Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-799cb7f958-9pt94/kube-storage-version-migrator-operator-799cb7f958-9pt94.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.357806288Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-799cb7f958-9pt94/kube-storage-version-migrator-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.357817608Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-799cb7f958-9pt94/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.357825948Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-799cb7f958-9pt94/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.357845829Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-799cb7f958-9pt94/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.357973543Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-799cb7f958-9pt94/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.358105047Z namespaces/openshift-kube-storage-version-migrator-operator/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.358120397Z namespaces/openshift-kube-storage-version-migrator-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.358182059Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.358244511Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.358326874Z namespaces/openshift-kube-storage-version-migrator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.358354344Z namespaces/openshift-kube-storage-version-migrator/openshift-kube-storage-version-migrator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.358423976Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.358464967Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.358516039Z namespaces/openshift-kube-storage-version-migrator/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.35854656Z namespaces/openshift-kube-storage-version-migrator/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.358616742Z namespaces/openshift-kube-storage-version-migrator/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.358742766Z namespaces/openshift-kube-storage-version-migrator/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.358840479Z namespaces/openshift-kube-storage-version-migrator/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.35888345Z namespaces/openshift-kube-storage-version-migrator/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.358937712Z namespaces/openshift-kube-storage-version-migrator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.358979833Z namespaces/openshift-kube-storage-version-migrator/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.359008394Z namespaces/openshift-kube-storage-version-migrator/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.359092646Z namespaces/openshift-kube-storage-version-migrator/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.359133947Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.359178919Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.359289152Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.359359934Z namespaces/openshift-kube-storage-version-migrator/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.359395855Z namespaces/openshift-kube-storage-version-migrator/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.359497168Z namespaces/openshift-kube-storage-version-migrator/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.359572751Z namespaces/openshift-kube-storage-version-migrator/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.359697804Z namespaces/openshift-kube-storage-version-migrator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.359775447Z namespaces/openshift-kube-storage-version-migrator/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.35989212Z namespaces/openshift-kube-storage-version-migrator/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.360013544Z namespaces/openshift-kube-storage-version-migrator/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.360165248Z namespaces/openshift-kube-storage-version-migrator/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.36020753Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.360287382Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.360339003Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.360371684Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.360427726Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.360467327Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.36054594Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.360626182Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.360677004Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.360713645Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.360765146Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.360800447Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.360852819Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.360859789Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.360915681Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.361073865Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.361281622Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.361430796Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.361630572Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.361818938Z namespaces/openshift-kube-storage-version-migrator/pods/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.361826068Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-bjfpf/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.361878669Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-bjfpf/migrator-7f59b54f6c-bjfpf.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.361952282Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-bjfpf/graceful-termination/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.361963432Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-bjfpf/graceful-termination/graceful-termination/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.361971372Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-bjfpf/graceful-termination/graceful-termination/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.361994243Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-bjfpf/graceful-termination/graceful-termination/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.362087496Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-bjfpf/graceful-termination/graceful-termination/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.362153248Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-bjfpf/graceful-termination/graceful-termination/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.362191669Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-bjfpf/migrator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.362201359Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-bjfpf/migrator/migrator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.362206929Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-bjfpf/migrator/migrator/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.362275411Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-bjfpf/migrator/migrator/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.362387425Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-bjfpf/migrator/migrator/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.362455187Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-bjfpf/migrator/migrator/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.362498238Z namespaces/openshift-kube-storage-version-migrator/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.362532959Z namespaces/openshift-kube-storage-version-migrator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.36258581Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.362621082Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.362674623Z namespaces/openshift-lws-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.362709244Z namespaces/openshift-lws-operator/openshift-lws-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.362780826Z namespaces/openshift-lws-operator/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.362811387Z namespaces/openshift-lws-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.362878339Z namespaces/openshift-lws-operator/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.36291404Z namespaces/openshift-lws-operator/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.363013673Z namespaces/openshift-lws-operator/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.363342813Z namespaces/openshift-lws-operator/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.363479807Z namespaces/openshift-lws-operator/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.363518768Z namespaces/openshift-lws-operator/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.36357619Z namespaces/openshift-lws-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.363631462Z namespaces/openshift-lws-operator/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.363662643Z namespaces/openshift-lws-operator/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.363752945Z namespaces/openshift-lws-operator/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.363792826Z namespaces/openshift-lws-operator/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.363839848Z namespaces/openshift-lws-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.36392044Z namespaces/openshift-lws-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.363956691Z namespaces/openshift-lws-operator/coordination.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.363966412Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.364024704Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/b8b2488c.x-k8s.io.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.364142837Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/openshift-lws-operator-lock.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.364204709Z namespaces/openshift-lws-operator/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.364282581Z namespaces/openshift-lws-operator/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.364412085Z namespaces/openshift-lws-operator/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.364503678Z namespaces/openshift-lws-operator/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.364828297Z namespaces/openshift-lws-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.36490575Z namespaces/openshift-lws-operator/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.365106726Z namespaces/openshift-lws-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.3652441Z namespaces/openshift-lws-operator/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.365480587Z namespaces/openshift-lws-operator/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.365553389Z namespaces/openshift-lws-operator/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.36559829Z namespaces/openshift-lws-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.365670402Z namespaces/openshift-lws-operator/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.365714924Z namespaces/openshift-lws-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.365764075Z namespaces/openshift-lws-operator/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.365804957Z namespaces/openshift-lws-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.365888079Z namespaces/openshift-lws-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.365965781Z namespaces/openshift-lws-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.366019633Z namespaces/openshift-lws-operator/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.366061354Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.366121616Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.366166057Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/lws-controller-manager-metrics-monitor.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.366219789Z namespaces/openshift-lws-operator/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.366288561Z namespaces/openshift-lws-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.366335782Z namespaces/openshift-lws-operator/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.366354143Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.366395764Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.366549829Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.366726364Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.366862218Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.367092055Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.367336462Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.367525768Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.367566519Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/install-rfpd9.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.367651862Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.367686043Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.367751635Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.367824607Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/leader-worker-set.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.367885409Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.367906739Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/leader-worker-set.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.367998332Z namespaces/openshift-lws-operator/pods/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.368010692Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5ff584b858-jl8gt/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.368041913Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5ff584b858-jl8gt/lws-controller-manager-5ff584b858-jl8gt.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.368142646Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5ff584b858-jl8gt/manager/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.368153997Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5ff584b858-jl8gt/manager/manager/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.368160097Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5ff584b858-jl8gt/manager/manager/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.368196188Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5ff584b858-jl8gt/manager/manager/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.36859899Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5ff584b858-jl8gt/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.368646431Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5ff584b858-jl8gt/manager/manager/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.368685772Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5ff584b858-xkf9w/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.368742804Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5ff584b858-xkf9w/lws-controller-manager-5ff584b858-xkf9w.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.368824277Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5ff584b858-xkf9w/manager/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.368834247Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5ff584b858-xkf9w/manager/manager/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.368840267Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5ff584b858-xkf9w/manager/manager/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.368881278Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5ff584b858-xkf9w/manager/manager/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.369005462Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5ff584b858-xkf9w/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.369079564Z namespaces/openshift-lws-operator/pods/lws-controller-manager-5ff584b858-xkf9w/manager/manager/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.369108305Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-mrl5g/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.369147436Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-mrl5g/openshift-lws-operator-fd8ccff4c-mrl5g.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.369247099Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-mrl5g/openshift-lws-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.36926094Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-mrl5g/openshift-lws-operator/openshift-lws-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.3692758Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-mrl5g/openshift-lws-operator/openshift-lws-operator/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.369339972Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-mrl5g/openshift-lws-operator/openshift-lws-operator/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.369525918Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-mrl5g/openshift-lws-operator/openshift-lws-operator/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.36959833Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-mrl5g/openshift-lws-operator/openshift-lws-operator/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.369634301Z namespaces/openshift-lws-operator/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.369660662Z namespaces/openshift-lws-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.369727674Z namespaces/openshift-lws-operator/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.369779345Z namespaces/openshift-lws-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.369835357Z namespaces/openshift-machine-api/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.369846417Z namespaces/openshift-machine-api/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.369853358Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.369896439Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.370076644Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.370300281Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.370448575Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.370671422Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.370903129Z namespaces/openshift-machine-config-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.370911059Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.370915029Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.37095275Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-controller.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.371057803Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-daemon.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.371135576Z namespaces/openshift-machine-config-operator/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.371145216Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.371179687Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.371392163Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.371579269Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.371719273Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.371932619Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.372119615Z namespaces/openshift-marketplace/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.372129625Z namespaces/openshift-marketplace/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.372137235Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.372174877Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/marketplace-alert-rules.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.372242639Z namespaces/openshift-marketplace/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.372259239Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.372340832Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/certified-operators.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.372468245Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/community-operators.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.372557188Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-marketplace.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.372666631Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-operators.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.372730203Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.372766424Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.372927079Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.373244199Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.373419184Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.37361111Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.373846877Z namespaces/openshift-monitoring/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.373874277Z namespaces/openshift-monitoring/openshift-monitoring.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.37395213Z namespaces/openshift-monitoring/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.373993471Z namespaces/openshift-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.374070023Z namespaces/openshift-monitoring/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.374116725Z namespaces/openshift-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.37430492Z namespaces/openshift-monitoring/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.374746633Z namespaces/openshift-monitoring/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.375113744Z namespaces/openshift-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.375381212Z namespaces/openshift-monitoring/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.375416464Z namespaces/openshift-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.375485555Z namespaces/openshift-monitoring/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.375516867Z namespaces/openshift-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.375609219Z namespaces/openshift-monitoring/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.375663641Z namespaces/openshift-monitoring/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.375707172Z namespaces/openshift-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.375791975Z namespaces/openshift-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.375893758Z namespaces/openshift-monitoring/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.375930519Z namespaces/openshift-monitoring/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.379211807Z namespaces/openshift-monitoring/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.379446604Z namespaces/openshift-monitoring/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.380544897Z namespaces/openshift-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.380626629Z namespaces/openshift-monitoring/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.381689851Z namespaces/openshift-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.381812674Z namespaces/openshift-monitoring/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.38231983Z namespaces/openshift-monitoring/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.382485064Z namespaces/openshift-monitoring/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.382510435Z namespaces/openshift-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.38267126Z namespaces/openshift-monitoring/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.382692561Z namespaces/openshift-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.382745172Z namespaces/openshift-monitoring/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.382790724Z namespaces/openshift-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.382873806Z namespaces/openshift-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.382950188Z namespaces/openshift-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.38299777Z namespaces/openshift-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.383041141Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.383178165Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.383209746Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/main.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.383314059Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.3833366Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/k8s.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.383466624Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.383488344Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/alertmanager-main-rules.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.383601228Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/cluster-monitoring-operator-prometheus-rules.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.383775853Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kube-state-metrics-rules.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.383864556Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kubernetes-monitoring-rules.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.384126893Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/node-exporter-rules.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.384282028Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-prometheus-rules.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.384428522Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-thanos-sidecar-rules.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.384515865Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-operator-rules.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.384635239Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/telemetry.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.384737451Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/thanos-querier.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.384823694Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.384879726Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/alertmanager-main.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.384971578Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/cluster-monitoring-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.385057251Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kube-state-metrics.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.385143174Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kubelet.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.385282958Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/metrics-server.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.38536657Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/node-exporter.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.385516315Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/openshift-state-metrics.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.385608848Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-k8s.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.385720191Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.385809334Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/telemeter-client.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.385909597Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-querier.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.386001879Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-sidecar.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.386076902Z namespaces/openshift-monitoring/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.386111372Z namespaces/openshift-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.386175574Z namespaces/openshift-monitoring/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.386182745Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.386215996Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.386474233Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.3867067Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.386851715Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.387053211Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.38735255Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.387401391Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/openshift-cluster-monitoring.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.387526645Z namespaces/openshift-monitoring/pods/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.387537175Z namespaces/openshift-monitoring/pods/alertmanager-main-0/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.387610087Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager-main-0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.387786053Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.387794793Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.387802973Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.387831244Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.387932987Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.387996729Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.3880407Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.388049001Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.388052961Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.388092172Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.388189875Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.388282677Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.388328169Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.388334149Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.388337859Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.388386711Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.388473083Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.388542375Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.388575746Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.388587386Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.388597357Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.388643738Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.388735851Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.388803303Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.388827553Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.388831634Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.388837524Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.388887115Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.388977098Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.38905085Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.389086951Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.389096912Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.389102892Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.389137223Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.389222115Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.389324968Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.389353639Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.38936178Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.38936843Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.389408441Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.389497233Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.389561266Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.389603647Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-whnln/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.389644578Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-whnln/cluster-monitoring-operator-5f7f494dd4-whnln.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.389741031Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-whnln/cluster-monitoring-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.389752121Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-whnln/cluster-monitoring-operator/cluster-monitoring-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.389756451Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-whnln/cluster-monitoring-operator/cluster-monitoring-operator/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.389790082Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-whnln/cluster-monitoring-operator/cluster-monitoring-operator/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.390666408Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-whnln/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.39073151Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-whnln/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.390772131Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8lv7g/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.390817253Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8lv7g/kube-state-metrics-785778ff74-8lv7g.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.390927236Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8lv7g/kube-rbac-proxy-main/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.390935487Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8lv7g/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.390941347Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8lv7g/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.390975278Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8lv7g/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.39106509Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8lv7g/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.391131302Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8lv7g/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.391177154Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8lv7g/kube-rbac-proxy-self/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.391184714Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8lv7g/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.391188654Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8lv7g/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.391222685Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8lv7g/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.391362039Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8lv7g/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.391411241Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8lv7g/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.391461072Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8lv7g/kube-state-metrics/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.391469983Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8lv7g/kube-state-metrics/kube-state-metrics/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.391473843Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8lv7g/kube-state-metrics/kube-state-metrics/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.391506364Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8lv7g/kube-state-metrics/kube-state-metrics/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.39171223Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8lv7g/kube-state-metrics/kube-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.391782672Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8lv7g/kube-state-metrics/kube-state-metrics/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.391822673Z namespaces/openshift-monitoring/pods/metrics-server-ffb8ff984-rft4x/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.391853654Z namespaces/openshift-monitoring/pods/metrics-server-ffb8ff984-rft4x/metrics-server-ffb8ff984-rft4x.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.391953087Z namespaces/openshift-monitoring/pods/metrics-server-ffb8ff984-rft4x/metrics-server/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.391960217Z namespaces/openshift-monitoring/pods/metrics-server-ffb8ff984-rft4x/metrics-server/metrics-server/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.391964547Z namespaces/openshift-monitoring/pods/metrics-server-ffb8ff984-rft4x/metrics-server/metrics-server/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.392004708Z namespaces/openshift-monitoring/pods/metrics-server-ffb8ff984-rft4x/metrics-server/metrics-server/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.392095951Z namespaces/openshift-monitoring/pods/metrics-server-ffb8ff984-rft4x/metrics-server/metrics-server/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.392164753Z namespaces/openshift-monitoring/pods/metrics-server-ffb8ff984-rft4x/metrics-server/metrics-server/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.392193974Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-jmgtf/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.392246966Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-jmgtf/monitoring-plugin-5f68df7f58-jmgtf.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.392365509Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-jmgtf/monitoring-plugin/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.39237623Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-jmgtf/monitoring-plugin/monitoring-plugin/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.39238202Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-jmgtf/monitoring-plugin/monitoring-plugin/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.392412711Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-jmgtf/monitoring-plugin/monitoring-plugin/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.392509254Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-jmgtf/monitoring-plugin/monitoring-plugin/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.392574496Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-jmgtf/monitoring-plugin/monitoring-plugin/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.392608557Z namespaces/openshift-monitoring/pods/node-exporter-cw9bv/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.392642858Z namespaces/openshift-monitoring/pods/node-exporter-cw9bv/node-exporter-cw9bv.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.392761371Z namespaces/openshift-monitoring/pods/node-exporter-cw9bv/init-textfile/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.392771361Z namespaces/openshift-monitoring/pods/node-exporter-cw9bv/init-textfile/init-textfile/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.392777051Z namespaces/openshift-monitoring/pods/node-exporter-cw9bv/init-textfile/init-textfile/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.392822453Z namespaces/openshift-monitoring/pods/node-exporter-cw9bv/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.392896635Z namespaces/openshift-monitoring/pods/node-exporter-cw9bv/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.392961497Z namespaces/openshift-monitoring/pods/node-exporter-cw9bv/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.392995498Z namespaces/openshift-monitoring/pods/node-exporter-cw9bv/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.393004998Z namespaces/openshift-monitoring/pods/node-exporter-cw9bv/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.393010869Z namespaces/openshift-monitoring/pods/node-exporter-cw9bv/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.39305298Z namespaces/openshift-monitoring/pods/node-exporter-cw9bv/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.393143502Z namespaces/openshift-monitoring/pods/node-exporter-cw9bv/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.393216715Z namespaces/openshift-monitoring/pods/node-exporter-cw9bv/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.393292017Z namespaces/openshift-monitoring/pods/node-exporter-cw9bv/node-exporter/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.393304337Z namespaces/openshift-monitoring/pods/node-exporter-cw9bv/node-exporter/node-exporter/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.393313987Z namespaces/openshift-monitoring/pods/node-exporter-cw9bv/node-exporter/node-exporter/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.393356709Z namespaces/openshift-monitoring/pods/node-exporter-cw9bv/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.393453562Z namespaces/openshift-monitoring/pods/node-exporter-cw9bv/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.393534644Z namespaces/openshift-monitoring/pods/node-exporter-cw9bv/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.393601906Z namespaces/openshift-monitoring/pods/node-exporter-hb676/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.393657298Z namespaces/openshift-monitoring/pods/node-exporter-hb676/node-exporter-hb676.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.393773341Z namespaces/openshift-monitoring/pods/node-exporter-hb676/init-textfile/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.393780652Z namespaces/openshift-monitoring/pods/node-exporter-hb676/init-textfile/init-textfile/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.393784532Z namespaces/openshift-monitoring/pods/node-exporter-hb676/init-textfile/init-textfile/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.393814413Z namespaces/openshift-monitoring/pods/node-exporter-hb676/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.393903965Z namespaces/openshift-monitoring/pods/node-exporter-hb676/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.393969707Z namespaces/openshift-monitoring/pods/node-exporter-hb676/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.394006398Z namespaces/openshift-monitoring/pods/node-exporter-hb676/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.394016239Z namespaces/openshift-monitoring/pods/node-exporter-hb676/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.394022339Z namespaces/openshift-monitoring/pods/node-exporter-hb676/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.3940612Z namespaces/openshift-monitoring/pods/node-exporter-hb676/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.394154183Z namespaces/openshift-monitoring/pods/node-exporter-hb676/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.394283036Z namespaces/openshift-monitoring/pods/node-exporter-hb676/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.394333798Z namespaces/openshift-monitoring/pods/node-exporter-hb676/node-exporter/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.394350749Z namespaces/openshift-monitoring/pods/node-exporter-hb676/node-exporter/node-exporter/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.394357299Z namespaces/openshift-monitoring/pods/node-exporter-hb676/node-exporter/node-exporter/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.39439147Z namespaces/openshift-monitoring/pods/node-exporter-hb676/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.394498613Z namespaces/openshift-monitoring/pods/node-exporter-hb676/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.394583045Z namespaces/openshift-monitoring/pods/node-exporter-hb676/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.394630717Z namespaces/openshift-monitoring/pods/node-exporter-s5sd7/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.394662498Z namespaces/openshift-monitoring/pods/node-exporter-s5sd7/node-exporter-s5sd7.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.394790462Z namespaces/openshift-monitoring/pods/node-exporter-s5sd7/init-textfile/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.394801072Z namespaces/openshift-monitoring/pods/node-exporter-s5sd7/init-textfile/init-textfile/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.394809742Z namespaces/openshift-monitoring/pods/node-exporter-s5sd7/init-textfile/init-textfile/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.394855113Z namespaces/openshift-monitoring/pods/node-exporter-s5sd7/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.394935286Z namespaces/openshift-monitoring/pods/node-exporter-s5sd7/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.39506976Z namespaces/openshift-monitoring/pods/node-exporter-s5sd7/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.39509482Z namespaces/openshift-monitoring/pods/node-exporter-s5sd7/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.395101591Z namespaces/openshift-monitoring/pods/node-exporter-s5sd7/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.395113781Z namespaces/openshift-monitoring/pods/node-exporter-s5sd7/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.395150752Z namespaces/openshift-monitoring/pods/node-exporter-s5sd7/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.395287246Z namespaces/openshift-monitoring/pods/node-exporter-s5sd7/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.395377599Z namespaces/openshift-monitoring/pods/node-exporter-s5sd7/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.395427701Z namespaces/openshift-monitoring/pods/node-exporter-s5sd7/node-exporter/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.395438541Z namespaces/openshift-monitoring/pods/node-exporter-s5sd7/node-exporter/node-exporter/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.395448831Z namespaces/openshift-monitoring/pods/node-exporter-s5sd7/node-exporter/node-exporter/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.395485532Z namespaces/openshift-monitoring/pods/node-exporter-s5sd7/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.395623446Z namespaces/openshift-monitoring/pods/node-exporter-s5sd7/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.395679498Z namespaces/openshift-monitoring/pods/node-exporter-s5sd7/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.395725359Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-dmh6c/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.395773531Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-dmh6c/openshift-state-metrics-84bd68b896-dmh6c.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.395931476Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-dmh6c/kube-rbac-proxy-main/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.395938586Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-dmh6c/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.395945136Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-dmh6c/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.395996278Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-dmh6c/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.39609021Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-dmh6c/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.396168493Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-dmh6c/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.396212894Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-dmh6c/kube-rbac-proxy-self/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.396220144Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-dmh6c/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.396254805Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-dmh6c/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.396308247Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-dmh6c/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.39642066Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-dmh6c/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.396496832Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-dmh6c/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.396537724Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-dmh6c/openshift-state-metrics/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.396547834Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-dmh6c/openshift-state-metrics/openshift-state-metrics/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.396553714Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-dmh6c/openshift-state-metrics/openshift-state-metrics/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.396597225Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-dmh6c/openshift-state-metrics/openshift-state-metrics/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.396716869Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-dmh6c/openshift-state-metrics/openshift-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.396785631Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-dmh6c/openshift-state-metrics/openshift-state-metrics/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.396835532Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.396915525Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus-k8s-0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.397115701Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.397124311Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.397128471Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.397170393Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.397312237Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.397386229Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.39743214Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.397442051Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.397448071Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.397491302Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.397587565Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.397657207Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.397707718Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.397717059Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.397724399Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.39776193Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.397855003Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.397935185Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.397974326Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.397982627Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.397986687Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.398018328Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.398122991Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.398194153Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.398240954Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.398252715Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.398256625Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.398314647Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.39841602Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.398488392Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.398506282Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.398513863Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.398522043Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.398565474Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.399187203Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.399289076Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.399314837Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.399324757Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.399330657Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.399395579Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.399490772Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.399561504Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.399578965Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-dbsvm/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.399645547Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-dbsvm/prometheus-operator-64785b6675-dbsvm.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.39974529Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-dbsvm/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.39975632Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-dbsvm/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.39976204Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-dbsvm/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.39977456Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-dbsvm/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.400042698Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-dbsvm/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.40010305Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-dbsvm/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.400123831Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-dbsvm/prometheus-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.400131951Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-dbsvm/prometheus-operator/prometheus-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.400137721Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-dbsvm/prometheus-operator/prometheus-operator/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.400188783Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-dbsvm/prometheus-operator/prometheus-operator/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.400500702Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-dbsvm/prometheus-operator/prometheus-operator/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.400552023Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-dbsvm/prometheus-operator/prometheus-operator/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.400575494Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-r78ll/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.400630806Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-r78ll/prometheus-operator-admission-webhook-77d98bc786-r78ll.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.40075646Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-r78ll/prometheus-operator-admission-webhook/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.40076442Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-r78ll/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.40076859Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-r78ll/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.400812951Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-r78ll/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.400906994Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-r78ll/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.400973796Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-r78ll/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.401005877Z namespaces/openshift-monitoring/pods/telemeter-client-5699db478d-mjnwl/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.401040858Z namespaces/openshift-monitoring/pods/telemeter-client-5699db478d-mjnwl/telemeter-client-5699db478d-mjnwl.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.401239604Z namespaces/openshift-monitoring/pods/telemeter-client-5699db478d-mjnwl/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.401275975Z namespaces/openshift-monitoring/pods/telemeter-client-5699db478d-mjnwl/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.401283025Z namespaces/openshift-monitoring/pods/telemeter-client-5699db478d-mjnwl/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.401307216Z namespaces/openshift-monitoring/pods/telemeter-client-5699db478d-mjnwl/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.401405459Z namespaces/openshift-monitoring/pods/telemeter-client-5699db478d-mjnwl/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.401473411Z namespaces/openshift-monitoring/pods/telemeter-client-5699db478d-mjnwl/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.401503882Z namespaces/openshift-monitoring/pods/telemeter-client-5699db478d-mjnwl/reload/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.401509972Z namespaces/openshift-monitoring/pods/telemeter-client-5699db478d-mjnwl/reload/reload/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.401516072Z namespaces/openshift-monitoring/pods/telemeter-client-5699db478d-mjnwl/reload/reload/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.401570104Z namespaces/openshift-monitoring/pods/telemeter-client-5699db478d-mjnwl/reload/reload/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.401664407Z namespaces/openshift-monitoring/pods/telemeter-client-5699db478d-mjnwl/reload/reload/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.401732739Z namespaces/openshift-monitoring/pods/telemeter-client-5699db478d-mjnwl/reload/reload/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.4017824Z namespaces/openshift-monitoring/pods/telemeter-client-5699db478d-mjnwl/telemeter-client/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.401797011Z namespaces/openshift-monitoring/pods/telemeter-client-5699db478d-mjnwl/telemeter-client/telemeter-client/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.401808471Z namespaces/openshift-monitoring/pods/telemeter-client-5699db478d-mjnwl/telemeter-client/telemeter-client/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.401852082Z namespaces/openshift-monitoring/pods/telemeter-client-5699db478d-mjnwl/telemeter-client/telemeter-client/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.401990796Z namespaces/openshift-monitoring/pods/telemeter-client-5699db478d-mjnwl/telemeter-client/telemeter-client/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.402062999Z namespaces/openshift-monitoring/pods/telemeter-client-5699db478d-mjnwl/telemeter-client/telemeter-client/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.402130821Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.402204033Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/thanos-querier-5d9d8df5d-6dwcd.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.402367297Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/kube-rbac-proxy-metrics/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.402382468Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.402388288Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.402418079Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.402645316Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.402711018Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.402820771Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/kube-rbac-proxy-rules/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.402830682Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/kube-rbac-proxy-rules/kube-rbac-proxy-rules/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.402836712Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.402878673Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.402968506Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.403036357Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.40310035Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/kube-rbac-proxy-web/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.40310765Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.40311148Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.403157931Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.403246314Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.403385488Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.403418479Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.403423219Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.403429279Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.403487161Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.403572904Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.403643856Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.403684267Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/prom-label-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.403693907Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.403719378Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.403749299Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.403823761Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.403891463Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.403930804Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/thanos-query/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.403941315Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/thanos-query/thanos-query/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.403945275Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/thanos-query/thanos-query/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.403973425Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/thanos-query/thanos-query/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.404074128Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/thanos-query/thanos-query/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.40413979Z namespaces/openshift-monitoring/pods/thanos-querier-5d9d8df5d-6dwcd/thanos-query/thanos-query/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.404169091Z namespaces/openshift-monitoring/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.404202372Z namespaces/openshift-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.404296775Z namespaces/openshift-monitoring/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.404343617Z namespaces/openshift-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.40444578Z namespaces/openshift-multus/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.404481891Z namespaces/openshift-multus/openshift-multus.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.404555183Z namespaces/openshift-multus/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.404590404Z namespaces/openshift-multus/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.404659516Z namespaces/openshift-multus/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.404692317Z namespaces/openshift-multus/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.404935604Z namespaces/openshift-multus/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.405013677Z namespaces/openshift-multus/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.4051151Z namespaces/openshift-multus/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.405150561Z namespaces/openshift-multus/apps/daemonsets/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.405201142Z namespaces/openshift-multus/apps/daemonsets/multus-additional-cni-plugins.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.405399338Z namespaces/openshift-multus/apps/daemonsets/multus.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.405527182Z namespaces/openshift-multus/apps/daemonsets/network-metrics-daemon.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.405595064Z namespaces/openshift-multus/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.405635485Z namespaces/openshift-multus/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.405681947Z namespaces/openshift-multus/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.405718808Z namespaces/openshift-multus/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.40580294Z namespaces/openshift-multus/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.405837691Z namespaces/openshift-multus/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.405905823Z namespaces/openshift-multus/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.405988706Z namespaces/openshift-multus/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.406043517Z namespaces/openshift-multus/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.406093269Z namespaces/openshift-multus/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.406245053Z namespaces/openshift-multus/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.406373087Z namespaces/openshift-multus/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.406925744Z namespaces/openshift-multus/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.407050887Z namespaces/openshift-multus/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.40781643Z namespaces/openshift-multus/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.407906873Z namespaces/openshift-multus/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.408042237Z namespaces/openshift-multus/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.408117289Z namespaces/openshift-multus/core/configmaps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.408167171Z namespaces/openshift-multus/core/configmaps/cni-copy-resources.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.408277434Z namespaces/openshift-multus/core/configmaps/default-cni-sysctl-allowlist.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.408397697Z namespaces/openshift-multus/core/configmaps/multus-daemon-config.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.40847588Z namespaces/openshift-multus/core/configmaps/whereabouts-flatfile-config.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.408573673Z namespaces/openshift-multus/core/serviceaccounts/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.408610134Z namespaces/openshift-multus/core/serviceaccounts/metrics-daemon-sa.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.408701357Z namespaces/openshift-multus/core/serviceaccounts/multus-ac.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.408783949Z namespaces/openshift-multus/core/serviceaccounts/multus-ancillary-tools.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.408871952Z namespaces/openshift-multus/core/serviceaccounts/multus.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.408912313Z namespaces/openshift-multus/core/services/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.408949214Z namespaces/openshift-multus/core/services/network-metrics-service.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.409016736Z namespaces/openshift-multus/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.409059907Z namespaces/openshift-multus/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.409112569Z namespaces/openshift-multus/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.40916169Z namespaces/openshift-multus/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.409214142Z namespaces/openshift-multus/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.409281124Z namespaces/openshift-multus/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.409377447Z namespaces/openshift-multus/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.409458319Z namespaces/openshift-multus/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.409524321Z namespaces/openshift-multus/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.409563802Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.409603313Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.409644225Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/monitor-network.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.409706097Z namespaces/openshift-multus/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.409734808Z namespaces/openshift-multus/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.409797349Z namespaces/openshift-multus/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.409804319Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.40983149Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.409999865Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.410183261Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.410368526Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.410575642Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.410806009Z namespaces/openshift-multus/pods/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.41081396Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.410857011Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/multus-additional-cni-plugins-2rdtb.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.411012816Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/bond-cni-plugin/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.411025236Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.411030066Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.411079238Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.411175551Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.411311415Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.411390387Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/cni-plugins/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.411407147Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/cni-plugins/cni-plugins/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.411412458Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.411455279Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.411546951Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.411649675Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.411706476Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/egress-router-binary-copy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.411715167Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.411721317Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.411759428Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.411850561Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.411923703Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.411939893Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.411949904Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.411955054Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.412002465Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.412075667Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.412139909Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.412187461Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/routeoverride-cni/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.412198061Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.412291274Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.412354996Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.412456269Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.412521271Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.412551792Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/whereabouts-cni-bincopy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.412556972Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.412560692Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.412636714Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.412731527Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.412799169Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.412814399Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/whereabouts-cni/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.41282276Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.41283242Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.412866601Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.412959214Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.413026746Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2rdtb/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.413055416Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.413109518Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/multus-additional-cni-plugins-bc97r.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.413280533Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/bond-cni-plugin/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.413291603Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.413295454Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.413326204Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.413419767Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.413482109Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.41352507Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/cni-plugins/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.413534541Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/cni-plugins/cni-plugins/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.413540381Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.413550401Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.413634154Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.413705956Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.413733407Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/egress-router-binary-copy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.413738627Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.413742287Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.413787018Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.413874201Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.413943003Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.413980784Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.413987684Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.413991564Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.414018155Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.414092178Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.414157809Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.41418647Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/routeoverride-cni/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.414201811Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.414206541Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.414243022Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.414341935Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.414421057Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.414436318Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/whereabouts-cni-bincopy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.414442848Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.414448668Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.41449307Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.414583352Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.414653574Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.414683495Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/whereabouts-cni/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.414687935Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.414691785Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.414731197Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.414814219Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.414887421Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bc97r/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.414916122Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.414969594Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/multus-additional-cni-plugins-kgrbw.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.415100817Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/bond-cni-plugin/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.415107948Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.415111828Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.415155389Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.415245372Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.415339915Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.415369165Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/cni-plugins/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.415373816Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/cni-plugins/cni-plugins/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.415377666Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.415434358Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.41551197Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.415577242Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.415642884Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/egress-router-binary-copy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.415652634Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.415662454Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.415683175Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.415783188Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.415831859Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.41586594Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.415886171Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.415893211Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.415901152Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.415972834Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.416042766Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.416076627Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/routeoverride-cni/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.416086337Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.416092107Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.416104978Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.416202341Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.416308114Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.416341125Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/whereabouts-cni-bincopy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.416347695Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.416353505Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.416409847Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.416504739Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.416572781Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.416608953Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/whereabouts-cni/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.416615703Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.416619483Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.416660964Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.416742337Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.416809629Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-kgrbw/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.41684448Z namespaces/openshift-multus/pods/multus-bgqcq/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.416873181Z namespaces/openshift-multus/pods/multus-bgqcq/multus-bgqcq.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.416975213Z namespaces/openshift-multus/pods/multus-bgqcq/kube-multus/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.416985184Z namespaces/openshift-multus/pods/multus-bgqcq/kube-multus/kube-multus/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.416991124Z namespaces/openshift-multus/pods/multus-bgqcq/kube-multus/kube-multus/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.417015615Z namespaces/openshift-multus/pods/multus-bgqcq/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.417988624Z namespaces/openshift-multus/pods/multus-bgqcq/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.418055466Z namespaces/openshift-multus/pods/multus-bgqcq/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.418094887Z namespaces/openshift-multus/pods/multus-bmsb4/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.418122318Z namespaces/openshift-multus/pods/multus-bmsb4/multus-bmsb4.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.418215631Z namespaces/openshift-multus/pods/multus-bmsb4/kube-multus/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.418239261Z namespaces/openshift-multus/pods/multus-bmsb4/kube-multus/kube-multus/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.418250542Z namespaces/openshift-multus/pods/multus-bmsb4/kube-multus/kube-multus/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.418295253Z namespaces/openshift-multus/pods/multus-bmsb4/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.418932042Z namespaces/openshift-multus/pods/multus-bmsb4/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.418993734Z namespaces/openshift-multus/pods/multus-bmsb4/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.419009104Z namespaces/openshift-multus/pods/multus-qtpwz/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.419066466Z namespaces/openshift-multus/pods/multus-qtpwz/multus-qtpwz.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.419157498Z namespaces/openshift-multus/pods/multus-qtpwz/kube-multus/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.419183829Z namespaces/openshift-multus/pods/multus-qtpwz/kube-multus/kube-multus/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.419189539Z namespaces/openshift-multus/pods/multus-qtpwz/kube-multus/kube-multus/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.41919959Z namespaces/openshift-multus/pods/multus-qtpwz/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.419951852Z namespaces/openshift-multus/pods/multus-qtpwz/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.420008454Z namespaces/openshift-multus/pods/multus-qtpwz/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.420034555Z namespaces/openshift-multus/pods/network-metrics-daemon-45fpx/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.420082346Z namespaces/openshift-multus/pods/network-metrics-daemon-45fpx/network-metrics-daemon-45fpx.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.420173289Z namespaces/openshift-multus/pods/network-metrics-daemon-45fpx/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.420180599Z namespaces/openshift-multus/pods/network-metrics-daemon-45fpx/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.420187209Z namespaces/openshift-multus/pods/network-metrics-daemon-45fpx/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.420239671Z namespaces/openshift-multus/pods/network-metrics-daemon-45fpx/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.420373535Z namespaces/openshift-multus/pods/network-metrics-daemon-45fpx/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.420436487Z namespaces/openshift-multus/pods/network-metrics-daemon-45fpx/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.420464768Z namespaces/openshift-multus/pods/network-metrics-daemon-45fpx/network-metrics-daemon/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.420471038Z namespaces/openshift-multus/pods/network-metrics-daemon-45fpx/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.420476878Z namespaces/openshift-multus/pods/network-metrics-daemon-45fpx/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.420524239Z namespaces/openshift-multus/pods/network-metrics-daemon-45fpx/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.420701785Z namespaces/openshift-multus/pods/network-metrics-daemon-45fpx/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.420769177Z namespaces/openshift-multus/pods/network-metrics-daemon-45fpx/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.420802598Z namespaces/openshift-multus/pods/network-metrics-daemon-4sl67/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.420851419Z namespaces/openshift-multus/pods/network-metrics-daemon-4sl67/network-metrics-daemon-4sl67.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.420930522Z namespaces/openshift-multus/pods/network-metrics-daemon-4sl67/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.420938192Z namespaces/openshift-multus/pods/network-metrics-daemon-4sl67/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.420942182Z namespaces/openshift-multus/pods/network-metrics-daemon-4sl67/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.420958372Z namespaces/openshift-multus/pods/network-metrics-daemon-4sl67/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.421065475Z namespaces/openshift-multus/pods/network-metrics-daemon-4sl67/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.421132237Z namespaces/openshift-multus/pods/network-metrics-daemon-4sl67/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.421156558Z namespaces/openshift-multus/pods/network-metrics-daemon-4sl67/network-metrics-daemon/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.421162578Z namespaces/openshift-multus/pods/network-metrics-daemon-4sl67/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.421168429Z namespaces/openshift-multus/pods/network-metrics-daemon-4sl67/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.42121364Z namespaces/openshift-multus/pods/network-metrics-daemon-4sl67/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.421402356Z namespaces/openshift-multus/pods/network-metrics-daemon-4sl67/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.421490008Z namespaces/openshift-multus/pods/network-metrics-daemon-4sl67/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.421523429Z namespaces/openshift-multus/pods/network-metrics-daemon-xnfv2/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.4215433Z namespaces/openshift-multus/pods/network-metrics-daemon-xnfv2/network-metrics-daemon-xnfv2.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.421675174Z namespaces/openshift-multus/pods/network-metrics-daemon-xnfv2/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.421687004Z namespaces/openshift-multus/pods/network-metrics-daemon-xnfv2/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.421692764Z namespaces/openshift-multus/pods/network-metrics-daemon-xnfv2/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.421702205Z namespaces/openshift-multus/pods/network-metrics-daemon-xnfv2/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.421788917Z namespaces/openshift-multus/pods/network-metrics-daemon-xnfv2/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.421859789Z namespaces/openshift-multus/pods/network-metrics-daemon-xnfv2/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.42187458Z namespaces/openshift-multus/pods/network-metrics-daemon-xnfv2/network-metrics-daemon/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.42188371Z namespaces/openshift-multus/pods/network-metrics-daemon-xnfv2/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.42188934Z namespaces/openshift-multus/pods/network-metrics-daemon-xnfv2/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.421926261Z namespaces/openshift-multus/pods/network-metrics-daemon-xnfv2/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.422098217Z namespaces/openshift-multus/pods/network-metrics-daemon-xnfv2/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.422169959Z namespaces/openshift-multus/pods/network-metrics-daemon-xnfv2/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.422187649Z namespaces/openshift-multus/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.42222325Z namespaces/openshift-multus/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.422336244Z namespaces/openshift-multus/rbac.authorization.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.422350784Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.422456597Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/multus-whereabouts.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.42254602Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.422596041Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.422642913Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.422730005Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/whereabouts-cni.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.422786897Z namespaces/openshift-multus/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.422833548Z namespaces/openshift-multus/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.42288031Z namespaces/openshift-must-gather-s6wcw/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.42288773Z namespaces/openshift-must-gather-s6wcw/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.42289591Z namespaces/openshift-must-gather-s6wcw/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.422941972Z namespaces/openshift-must-gather-s6wcw/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.423102136Z namespaces/openshift-must-gather-s6wcw/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.423308753Z namespaces/openshift-must-gather-s6wcw/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.423459667Z namespaces/openshift-must-gather-s6wcw/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.423669163Z namespaces/openshift-must-gather-s6wcw/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.42388163Z namespaces/openshift-must-gather-stwlp/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.42389191Z namespaces/openshift-must-gather-stwlp/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.42389771Z namespaces/openshift-must-gather-stwlp/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.42390676Z namespaces/openshift-must-gather-stwlp/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.424148508Z namespaces/openshift-must-gather-stwlp/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.424389215Z namespaces/openshift-must-gather-stwlp/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.424688394Z namespaces/openshift-must-gather-stwlp/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.42488122Z namespaces/openshift-must-gather-stwlp/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.425073555Z namespaces/openshift-network-console/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.425109656Z namespaces/openshift-network-console/openshift-network-console.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.425180538Z namespaces/openshift-network-console/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.425199509Z namespaces/openshift-network-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.425298352Z namespaces/openshift-network-console/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.425363294Z namespaces/openshift-network-console/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.425454917Z namespaces/openshift-network-console/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.42556091Z namespaces/openshift-network-console/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.425671053Z namespaces/openshift-network-console/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.425724554Z namespaces/openshift-network-console/apps/deployments/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.425754106Z namespaces/openshift-network-console/apps/deployments/networking-console-plugin.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.425827138Z namespaces/openshift-network-console/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.425870109Z namespaces/openshift-network-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.42591392Z namespaces/openshift-network-console/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.425961362Z namespaces/openshift-network-console/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.426040414Z namespaces/openshift-network-console/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.426104896Z namespaces/openshift-network-console/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.426160538Z namespaces/openshift-network-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.426260661Z namespaces/openshift-network-console/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.426332083Z namespaces/openshift-network-console/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.426370054Z namespaces/openshift-network-console/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.426487537Z namespaces/openshift-network-console/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.42656899Z namespaces/openshift-network-console/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.426699484Z namespaces/openshift-network-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.426770376Z namespaces/openshift-network-console/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.42691898Z namespaces/openshift-network-console/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.427046124Z namespaces/openshift-network-console/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.427164877Z namespaces/openshift-network-console/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.427201429Z namespaces/openshift-network-console/core/configmaps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.427303742Z namespaces/openshift-network-console/core/configmaps/networking-console-plugin.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.427367194Z namespaces/openshift-network-console/core/services/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.427400045Z namespaces/openshift-network-console/core/services/networking-console-plugin.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.427462877Z namespaces/openshift-network-console/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.427487147Z namespaces/openshift-network-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.427560319Z namespaces/openshift-network-console/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.42757609Z namespaces/openshift-network-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.427640642Z namespaces/openshift-network-console/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.427682763Z namespaces/openshift-network-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.427762706Z namespaces/openshift-network-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.427840758Z namespaces/openshift-network-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.427876089Z namespaces/openshift-network-console/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.42790961Z namespaces/openshift-network-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.427961041Z namespaces/openshift-network-console/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.427993272Z namespaces/openshift-network-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.428029103Z namespaces/openshift-network-console/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.428037994Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.428084325Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.42825303Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.428491057Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.428646622Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.428840128Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.429024153Z namespaces/openshift-network-console/pods/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.429034573Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-bdj79/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.429048824Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-bdj79/networking-console-plugin-6cf856758c-bdj79.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.429153087Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-bdj79/networking-console-plugin/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.429164347Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-bdj79/networking-console-plugin/networking-console-plugin/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.429177788Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-bdj79/networking-console-plugin/networking-console-plugin/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.429186768Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-bdj79/networking-console-plugin/networking-console-plugin/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.429309272Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-bdj79/networking-console-plugin/networking-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.429392504Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-bdj79/networking-console-plugin/networking-console-plugin/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.429413455Z namespaces/openshift-network-console/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.429442906Z namespaces/openshift-network-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.429512388Z namespaces/openshift-network-console/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.429530428Z namespaces/openshift-network-console/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.4295978Z namespaces/openshift-network-diagnostics/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.429618891Z namespaces/openshift-network-diagnostics/openshift-network-diagnostics.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.429683283Z namespaces/openshift-network-diagnostics/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.429715904Z namespaces/openshift-network-diagnostics/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.429773626Z namespaces/openshift-network-diagnostics/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.429805796Z namespaces/openshift-network-diagnostics/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.42991114Z namespaces/openshift-network-diagnostics/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.430018223Z namespaces/openshift-network-diagnostics/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.430114506Z namespaces/openshift-network-diagnostics/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.430144486Z namespaces/openshift-network-diagnostics/apps/daemonsets/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.430185728Z namespaces/openshift-network-diagnostics/apps/daemonsets/network-check-target.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.430283141Z namespaces/openshift-network-diagnostics/apps/deployments/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.430319642Z namespaces/openshift-network-diagnostics/apps/deployments/network-check-source.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.430375924Z namespaces/openshift-network-diagnostics/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.430416515Z namespaces/openshift-network-diagnostics/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.430475456Z namespaces/openshift-network-diagnostics/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.430501067Z namespaces/openshift-network-diagnostics/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.430567069Z namespaces/openshift-network-diagnostics/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.430619201Z namespaces/openshift-network-diagnostics/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.430660252Z namespaces/openshift-network-diagnostics/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.430740684Z namespaces/openshift-network-diagnostics/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.430798646Z namespaces/openshift-network-diagnostics/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.430839167Z namespaces/openshift-network-diagnostics/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.431116186Z namespaces/openshift-network-diagnostics/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.431206178Z namespaces/openshift-network-diagnostics/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.431428225Z namespaces/openshift-network-diagnostics/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.431514597Z namespaces/openshift-network-diagnostics/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.431715983Z namespaces/openshift-network-diagnostics/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.431801676Z namespaces/openshift-network-diagnostics/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.431911979Z namespaces/openshift-network-diagnostics/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.431967261Z namespaces/openshift-network-diagnostics/core/serviceaccounts/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.432002752Z namespaces/openshift-network-diagnostics/core/serviceaccounts/network-diagnostics.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.432059784Z namespaces/openshift-network-diagnostics/core/services/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.432093445Z namespaces/openshift-network-diagnostics/core/services/network-check-source.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.432182577Z namespaces/openshift-network-diagnostics/core/services/network-check-target.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.432241489Z namespaces/openshift-network-diagnostics/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.43229185Z namespaces/openshift-network-diagnostics/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.432376913Z namespaces/openshift-network-diagnostics/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.432407314Z namespaces/openshift-network-diagnostics/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.432462406Z namespaces/openshift-network-diagnostics/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.432498447Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.432584329Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.432664432Z namespaces/openshift-network-diagnostics/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.432717923Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.432740214Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.432806436Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.432833627Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/network-check-source.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.432894528Z namespaces/openshift-network-diagnostics/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.432921709Z namespaces/openshift-network-diagnostics/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.432975911Z namespaces/openshift-network-diagnostics/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.432983551Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.433025003Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.433171317Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.433378023Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.433516447Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.433722643Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.433910729Z namespaces/openshift-network-diagnostics/pods/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.433918099Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-jfqf9/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.43393905Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-jfqf9/network-check-source-869dcfcdcd-jfqf9.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.434023002Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-jfqf9/check-endpoints/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.434030602Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-jfqf9/check-endpoints/check-endpoints/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.434043913Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-jfqf9/check-endpoints/check-endpoints/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.434050653Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-jfqf9/check-endpoints/check-endpoints/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.434173487Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-jfqf9/check-endpoints/check-endpoints/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.434249109Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-jfqf9/check-endpoints/check-endpoints/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.434304981Z namespaces/openshift-network-diagnostics/pods/network-check-target-5l5l6/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.434347972Z namespaces/openshift-network-diagnostics/pods/network-check-target-5l5l6/network-check-target-5l5l6.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.434430134Z namespaces/openshift-network-diagnostics/pods/network-check-target-5l5l6/network-check-target-container/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.434443145Z namespaces/openshift-network-diagnostics/pods/network-check-target-5l5l6/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.434449115Z namespaces/openshift-network-diagnostics/pods/network-check-target-5l5l6/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.434501916Z namespaces/openshift-network-diagnostics/pods/network-check-target-5l5l6/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.43461859Z namespaces/openshift-network-diagnostics/pods/network-check-target-5l5l6/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.434719773Z namespaces/openshift-network-diagnostics/pods/network-check-target-5l5l6/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.434755684Z namespaces/openshift-network-diagnostics/pods/network-check-target-5tgm8/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.434796395Z namespaces/openshift-network-diagnostics/pods/network-check-target-5tgm8/network-check-target-5tgm8.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.434878088Z namespaces/openshift-network-diagnostics/pods/network-check-target-5tgm8/network-check-target-container/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.434887658Z namespaces/openshift-network-diagnostics/pods/network-check-target-5tgm8/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.434893748Z namespaces/openshift-network-diagnostics/pods/network-check-target-5tgm8/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.434921519Z namespaces/openshift-network-diagnostics/pods/network-check-target-5tgm8/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.435006052Z namespaces/openshift-network-diagnostics/pods/network-check-target-5tgm8/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.435075414Z namespaces/openshift-network-diagnostics/pods/network-check-target-5tgm8/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.435112005Z namespaces/openshift-network-diagnostics/pods/network-check-target-xrtxq/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.435137586Z namespaces/openshift-network-diagnostics/pods/network-check-target-xrtxq/network-check-target-xrtxq.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.435218218Z namespaces/openshift-network-diagnostics/pods/network-check-target-xrtxq/network-check-target-container/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.435240049Z namespaces/openshift-network-diagnostics/pods/network-check-target-xrtxq/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.435248489Z namespaces/openshift-network-diagnostics/pods/network-check-target-xrtxq/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.435367513Z namespaces/openshift-network-diagnostics/pods/network-check-target-xrtxq/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.435443045Z namespaces/openshift-network-diagnostics/pods/network-check-target-xrtxq/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.435513897Z namespaces/openshift-network-diagnostics/pods/network-check-target-xrtxq/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.435539378Z namespaces/openshift-network-diagnostics/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.435588719Z namespaces/openshift-network-diagnostics/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.43564091Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.435647951Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.435679302Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.435757514Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.435812296Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.435846817Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/network-diagnostics.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.435927939Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.435974931Z namespaces/openshift-network-diagnostics/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.436022152Z namespaces/openshift-network-diagnostics/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.436060093Z namespaces/openshift-network-node-identity/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.436109164Z namespaces/openshift-network-node-identity/openshift-network-node-identity.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.436167016Z namespaces/openshift-network-node-identity/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.436211108Z namespaces/openshift-network-node-identity/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.43627959Z namespaces/openshift-network-node-identity/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.436345582Z namespaces/openshift-network-node-identity/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.436416444Z namespaces/openshift-network-node-identity/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.436501586Z namespaces/openshift-network-node-identity/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.436577089Z namespaces/openshift-network-node-identity/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.43663897Z namespaces/openshift-network-node-identity/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.436665101Z namespaces/openshift-network-node-identity/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.436721403Z namespaces/openshift-network-node-identity/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.436753944Z namespaces/openshift-network-node-identity/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.436835456Z namespaces/openshift-network-node-identity/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.436878067Z namespaces/openshift-network-node-identity/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.436913328Z namespaces/openshift-network-node-identity/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.437000541Z namespaces/openshift-network-node-identity/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.437042522Z namespaces/openshift-network-node-identity/coordination.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.437048783Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.437094404Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ovnkube-identity.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.437128775Z namespaces/openshift-network-node-identity/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.437178456Z namespaces/openshift-network-node-identity/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.43729994Z namespaces/openshift-network-node-identity/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.437378212Z namespaces/openshift-network-node-identity/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.437475625Z namespaces/openshift-network-node-identity/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.437542747Z namespaces/openshift-network-node-identity/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.43762383Z namespaces/openshift-network-node-identity/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.437770114Z namespaces/openshift-network-node-identity/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.437882667Z namespaces/openshift-network-node-identity/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.437926739Z namespaces/openshift-network-node-identity/core/serviceaccounts/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.43795713Z namespaces/openshift-network-node-identity/core/serviceaccounts/network-node-identity.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.438018421Z namespaces/openshift-network-node-identity/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.438058403Z namespaces/openshift-network-node-identity/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.438111894Z namespaces/openshift-network-node-identity/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.438130535Z namespaces/openshift-network-node-identity/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.438196667Z namespaces/openshift-network-node-identity/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.438221227Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.438340131Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.438420413Z namespaces/openshift-network-node-identity/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.438464275Z namespaces/openshift-network-node-identity/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.438485705Z namespaces/openshift-network-node-identity/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.438549687Z namespaces/openshift-network-node-identity/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.438589689Z namespaces/openshift-network-node-identity/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.43863436Z namespaces/openshift-network-node-identity/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.43864148Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.438679131Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.43897814Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.439160746Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.43932061Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.439518726Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.439713632Z namespaces/openshift-network-node-identity/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.439745383Z namespaces/openshift-network-node-identity/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.439798905Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.439807575Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.439845516Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/network-node-identity-leases.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.439878587Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.439930578Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/network-node-identity-leases.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.43997742Z namespaces/openshift-network-node-identity/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.440017611Z namespaces/openshift-network-node-identity/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.440063422Z namespaces/openshift-network-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.440113264Z namespaces/openshift-network-operator/openshift-network-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.440171406Z namespaces/openshift-network-operator/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.440212377Z namespaces/openshift-network-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.440384122Z namespaces/openshift-network-operator/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.440417493Z namespaces/openshift-network-operator/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.440546147Z namespaces/openshift-network-operator/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.440624789Z namespaces/openshift-network-operator/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.440707102Z namespaces/openshift-network-operator/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.440745653Z namespaces/openshift-network-operator/apps/daemonsets/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.440784134Z namespaces/openshift-network-operator/apps/daemonsets/iptables-alerter.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.440857356Z namespaces/openshift-network-operator/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.440881627Z namespaces/openshift-network-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.440942339Z namespaces/openshift-network-operator/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.44097791Z namespaces/openshift-network-operator/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.441060832Z namespaces/openshift-network-operator/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.441112504Z namespaces/openshift-network-operator/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.441141345Z namespaces/openshift-network-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.441250208Z namespaces/openshift-network-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.44130706Z namespaces/openshift-network-operator/coordination.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.44131407Z namespaces/openshift-network-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.441359091Z namespaces/openshift-network-operator/coordination.k8s.io/leases/network-operator-lock.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.441411463Z namespaces/openshift-network-operator/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.441445894Z namespaces/openshift-network-operator/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.441582938Z namespaces/openshift-network-operator/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.44166318Z namespaces/openshift-network-operator/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.441899137Z namespaces/openshift-network-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.44197756Z namespaces/openshift-network-operator/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.442173145Z namespaces/openshift-network-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.442298239Z namespaces/openshift-network-operator/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.442429193Z namespaces/openshift-network-operator/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.442477385Z namespaces/openshift-network-operator/core/configmaps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.442517976Z namespaces/openshift-network-operator/core/configmaps/applied-cluster.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.44264874Z namespaces/openshift-network-operator/core/configmaps/iptables-alerter-script.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.442728402Z namespaces/openshift-network-operator/core/serviceaccounts/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.442761993Z namespaces/openshift-network-operator/core/serviceaccounts/iptables-alerter.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.442821215Z namespaces/openshift-network-operator/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.442844125Z namespaces/openshift-network-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.442907807Z namespaces/openshift-network-operator/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.442954279Z namespaces/openshift-network-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.443019021Z namespaces/openshift-network-operator/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.443058552Z namespaces/openshift-network-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.443143044Z namespaces/openshift-network-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.443248048Z namespaces/openshift-network-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.443311109Z namespaces/openshift-network-operator/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.443347351Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.443414313Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.443448603Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/openshift-network-operator-ipsec-rules.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.443512865Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.443566607Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/network-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.443621929Z namespaces/openshift-network-operator/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.44365599Z namespaces/openshift-network-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.443704911Z namespaces/openshift-network-operator/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.443712121Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.443760093Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.443921857Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.444118334Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.444283288Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.444516155Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.444719692Z namespaces/openshift-network-operator/pods/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.444727982Z namespaces/openshift-network-operator/pods/iptables-alerter-54jpc/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.444770583Z namespaces/openshift-network-operator/pods/iptables-alerter-54jpc/iptables-alerter-54jpc.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.444825355Z namespaces/openshift-network-operator/pods/iptables-alerter-54jpc/iptables-alerter/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.444833105Z namespaces/openshift-network-operator/pods/iptables-alerter-54jpc/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.444843475Z namespaces/openshift-network-operator/pods/iptables-alerter-54jpc/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.444884486Z namespaces/openshift-network-operator/pods/iptables-alerter-54jpc/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.444970089Z namespaces/openshift-network-operator/pods/iptables-alerter-54jpc/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.445047731Z namespaces/openshift-network-operator/pods/iptables-alerter-54jpc/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.445065502Z namespaces/openshift-network-operator/pods/iptables-alerter-j7qjq/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.445116783Z namespaces/openshift-network-operator/pods/iptables-alerter-j7qjq/iptables-alerter-j7qjq.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.445189395Z namespaces/openshift-network-operator/pods/iptables-alerter-j7qjq/iptables-alerter/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.445196716Z namespaces/openshift-network-operator/pods/iptables-alerter-j7qjq/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.445201476Z namespaces/openshift-network-operator/pods/iptables-alerter-j7qjq/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.445238937Z namespaces/openshift-network-operator/pods/iptables-alerter-j7qjq/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.44535373Z namespaces/openshift-network-operator/pods/iptables-alerter-j7qjq/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.445424253Z namespaces/openshift-network-operator/pods/iptables-alerter-j7qjq/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.445462994Z namespaces/openshift-network-operator/pods/iptables-alerter-pxrrr/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.445502085Z namespaces/openshift-network-operator/pods/iptables-alerter-pxrrr/iptables-alerter-pxrrr.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.445576187Z namespaces/openshift-network-operator/pods/iptables-alerter-pxrrr/iptables-alerter/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.445585557Z namespaces/openshift-network-operator/pods/iptables-alerter-pxrrr/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.445590538Z namespaces/openshift-network-operator/pods/iptables-alerter-pxrrr/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.445601628Z namespaces/openshift-network-operator/pods/iptables-alerter-pxrrr/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.445702501Z namespaces/openshift-network-operator/pods/iptables-alerter-pxrrr/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.445772403Z namespaces/openshift-network-operator/pods/iptables-alerter-pxrrr/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.445814644Z namespaces/openshift-network-operator/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.445849795Z namespaces/openshift-network-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.445897647Z namespaces/openshift-network-operator/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.445929308Z namespaces/openshift-network-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.445990189Z namespaces/openshift-node/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.4459998Z namespaces/openshift-node/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.44600543Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.446039621Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.446182095Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.446391651Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.446530375Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.446734121Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.447097432Z namespaces/openshift-operator-lifecycle-manager/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.447144914Z namespaces/openshift-operator-lifecycle-manager/openshift-operator-lifecycle-manager.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.447207386Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.447269557Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.447325609Z namespaces/openshift-operator-lifecycle-manager/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.447367391Z namespaces/openshift-operator-lifecycle-manager/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.447446853Z namespaces/openshift-operator-lifecycle-manager/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.447544086Z namespaces/openshift-operator-lifecycle-manager/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.447623018Z namespaces/openshift-operator-lifecycle-manager/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.44767232Z namespaces/openshift-operator-lifecycle-manager/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.44770533Z namespaces/openshift-operator-lifecycle-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.447755352Z namespaces/openshift-operator-lifecycle-manager/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.447798423Z namespaces/openshift-operator-lifecycle-manager/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.447873646Z namespaces/openshift-operator-lifecycle-manager/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.447935768Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.447975849Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.448059391Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.448102642Z namespaces/openshift-operator-lifecycle-manager/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.448145374Z namespaces/openshift-operator-lifecycle-manager/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.448250007Z namespaces/openshift-operator-lifecycle-manager/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.448375941Z namespaces/openshift-operator-lifecycle-manager/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.448481014Z namespaces/openshift-operator-lifecycle-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.448559706Z namespaces/openshift-operator-lifecycle-manager/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.448640429Z namespaces/openshift-operator-lifecycle-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.448777153Z namespaces/openshift-operator-lifecycle-manager/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.448967768Z namespaces/openshift-operator-lifecycle-manager/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.44903475Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.449063261Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.449130973Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.449183025Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.449236856Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.449306078Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.449406991Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.449487854Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.449531125Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.449579697Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.449636718Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.449680589Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/olm-alert-rules.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.449736881Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.449779522Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.449867795Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.449875765Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.449914687Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.450080861Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.450329459Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.450474403Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.450672979Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.450858155Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.450887705Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/olm-operators.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.450952747Z namespaces/openshift-operator-lifecycle-manager/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.450991599Z namespaces/openshift-operator-lifecycle-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.45104619Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.451074201Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.451131913Z namespaces/openshift-operators/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.451139043Z namespaces/openshift-operators/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.451142983Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.451176614Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.45135429Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.451533505Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.451679839Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.451872475Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.452147213Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.452186764Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/global-operators.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.452251746Z namespaces/openshift-ovn-kubernetes/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.452310818Z namespaces/openshift-ovn-kubernetes/openshift-ovn-kubernetes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.45238105Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.452419841Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.452477813Z namespaces/openshift-ovn-kubernetes/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.452501104Z namespaces/openshift-ovn-kubernetes/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.45269643Z namespaces/openshift-ovn-kubernetes/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.452774682Z namespaces/openshift-ovn-kubernetes/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.452854644Z namespaces/openshift-ovn-kubernetes/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.452904176Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.452938407Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ovnkube-node.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.453077541Z namespaces/openshift-ovn-kubernetes/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.453092201Z namespaces/openshift-ovn-kubernetes/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.453147313Z namespaces/openshift-ovn-kubernetes/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.453187904Z namespaces/openshift-ovn-kubernetes/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.453296468Z namespaces/openshift-ovn-kubernetes/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.453346319Z namespaces/openshift-ovn-kubernetes/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.45338068Z namespaces/openshift-ovn-kubernetes/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.453463322Z namespaces/openshift-ovn-kubernetes/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.453500294Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.453508184Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.453552825Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ovn-kubernetes-master.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.453605107Z namespaces/openshift-ovn-kubernetes/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.453648568Z namespaces/openshift-ovn-kubernetes/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.453897055Z namespaces/openshift-ovn-kubernetes/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.453988998Z namespaces/openshift-ovn-kubernetes/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.454341859Z namespaces/openshift-ovn-kubernetes/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.454481343Z namespaces/openshift-ovn-kubernetes/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.454987288Z namespaces/openshift-ovn-kubernetes/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.45507479Z namespaces/openshift-ovn-kubernetes/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.455241245Z namespaces/openshift-ovn-kubernetes/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.45538941Z namespaces/openshift-ovn-kubernetes/core/configmaps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.455433811Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-config.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.455522614Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-script-lib.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.455656728Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.455694669Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-control-plane.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.455785602Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-node.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.455833433Z namespaces/openshift-ovn-kubernetes/core/services/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.455875324Z namespaces/openshift-ovn-kubernetes/core/services/ovn-kubernetes-node.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.455943656Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.455979737Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.456039259Z namespaces/openshift-ovn-kubernetes/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.45606712Z namespaces/openshift-ovn-kubernetes/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.456126752Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.456133892Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.456172733Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/default.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.456235535Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.456288277Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.456386749Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.456467882Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.456513353Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.456540934Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.456611956Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.456628487Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/networking-rules.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.4567336Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.45674665Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/monitor-ovn-node.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.456814202Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.456821482Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.456841643Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ovn.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.456937896Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/signer.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.456987487Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.457028359Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.45707481Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.45708288Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.457120611Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.457429831Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.457616866Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.457781291Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.457985107Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.458184423Z namespaces/openshift-ovn-kubernetes/pods/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.458191943Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.458269356Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/ovnkube-node-2bscc.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.458441601Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/kube-rbac-proxy-node/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.458449411Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.458453321Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.458488702Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.458618736Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.458690468Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.458712339Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.458719129Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.458724339Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.458771461Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.458878884Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.458942556Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.458977717Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/nbdb/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.459001408Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/nbdb/nbdb/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.459010508Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/nbdb/nbdb/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.459019798Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/nbdb/nbdb/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.459140432Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.459210034Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.459249455Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/northd/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.459264115Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/northd/northd/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.459268916Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/northd/northd/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.459303847Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/northd/northd/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.45942333Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.459496642Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/northd/northd/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.459515323Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/ovn-acl-logging/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.459521493Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.459533043Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.459594295Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.459679998Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.4597363Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/ovn-controller/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.45975822Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/ovn-controller/ovn-controller/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.45976225Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.459787551Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.46010052Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.460183963Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.460216604Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/ovnkube-controller/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.460221164Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.460239805Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.460291016Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.461908994Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.461976296Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.461994627Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/sbdb/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.461999557Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/sbdb/sbdb/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.462019748Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/sbdb/sbdb/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.462048058Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/sbdb/sbdb/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.462154532Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.462236784Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-2bscc/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.462254215Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.462328937Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/ovnkube-node-c2s8k.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.462488392Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/kube-rbac-proxy-node/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.462497252Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.462502942Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.462529213Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.462659007Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.462726409Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.4627707Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.462782611Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.462788561Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.462801381Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.462926605Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.462997587Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.463013137Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/nbdb/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.463021878Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/nbdb/nbdb/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.463025738Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/nbdb/nbdb/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.463078929Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/nbdb/nbdb/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.463180272Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.463266695Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.463283985Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/northd/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.463291546Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/northd/northd/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.463296206Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/northd/northd/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.463369788Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/northd/northd/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.463476611Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.463544093Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/northd/northd/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.463558203Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/ovn-acl-logging/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.463566054Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.463587335Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.463608605Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.463705398Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.46376626Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/ovn-controller/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.46377333Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/ovn-controller/ovn-controller/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.463791581Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.463812251Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.464173722Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.464249824Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.464331967Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/ovnkube-controller/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.464340887Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.464344917Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.464367858Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.466331997Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.466402559Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.466429949Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/sbdb/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.466435209Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/sbdb/sbdb/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.46643908Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/sbdb/sbdb/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.466478451Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/sbdb/sbdb/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.466578174Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.466649036Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-c2s8k/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.466685137Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.466757949Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/ovnkube-node-v76kh.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.466918564Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/kube-rbac-proxy-node/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.466928964Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.466938035Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.466947075Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.467084529Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.467153141Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.467174491Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.467180012Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.467185602Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.467258224Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.467421039Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.467486671Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.467512172Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/nbdb/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.467520442Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/nbdb/nbdb/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.467525122Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/nbdb/nbdb/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.467566333Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/nbdb/nbdb/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.467664206Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.467730588Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.46778548Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/northd/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.46779311Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/northd/northd/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.46779704Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/northd/northd/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.467835631Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/northd/northd/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.467943104Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.468011047Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/northd/northd/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.468062088Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/ovn-acl-logging/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.468069268Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.468080929Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.468102069Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.468198962Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.468275504Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/ovn-controller/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.468297055Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/ovn-controller/ovn-controller/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.468301585Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.468334986Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.468658716Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.468744988Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.468763209Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/ovnkube-controller/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.468770309Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.468774179Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.468825391Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.470525822Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.470581483Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.470627685Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/sbdb/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.470639595Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/sbdb/sbdb/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.470644425Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/sbdb/sbdb/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.470665266Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/sbdb/sbdb/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.470766089Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.470831091Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v76kh/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.470856171Z namespaces/openshift-ovn-kubernetes/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.470904423Z namespaces/openshift-ovn-kubernetes/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.470957355Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.470966515Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.470995325Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.471091299Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-nodes-identity-limited.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.471195131Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.471281574Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.471326335Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.471418678Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.471502011Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.471539592Z namespaces/openshift-ovn-kubernetes/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.471584983Z namespaces/openshift-ovn-kubernetes/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.471638605Z namespaces/openshift-route-controller-manager/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.471647975Z namespaces/openshift-route-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.471654105Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.471679406Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/openshift-route-controllers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.471736558Z namespaces/openshift-route-controller-manager/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.471744578Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.471761899Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.471975995Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.472201962Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.472384597Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.472596223Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.4728105Z namespaces/openshift-service-ca-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.472847601Z namespaces/openshift-service-ca-operator/openshift-service-ca-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.472906993Z namespaces/openshift-service-ca-operator/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.472937774Z namespaces/openshift-service-ca-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.472997835Z namespaces/openshift-service-ca-operator/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.473030096Z namespaces/openshift-service-ca-operator/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.473111479Z namespaces/openshift-service-ca-operator/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.473242052Z namespaces/openshift-service-ca-operator/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.473344546Z namespaces/openshift-service-ca-operator/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.473382317Z namespaces/openshift-service-ca-operator/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.473429318Z namespaces/openshift-service-ca-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.473472569Z namespaces/openshift-service-ca-operator/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.473503Z namespaces/openshift-service-ca-operator/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.473593553Z namespaces/openshift-service-ca-operator/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.473630324Z namespaces/openshift-service-ca-operator/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.473679396Z namespaces/openshift-service-ca-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.473768718Z namespaces/openshift-service-ca-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.473813Z namespaces/openshift-service-ca-operator/coordination.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.47381978Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.473862561Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/service-ca-operator-lock.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.473915173Z namespaces/openshift-service-ca-operator/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.473955564Z namespaces/openshift-service-ca-operator/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.474067047Z namespaces/openshift-service-ca-operator/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.47415004Z namespaces/openshift-service-ca-operator/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.474403587Z namespaces/openshift-service-ca-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.47448266Z namespaces/openshift-service-ca-operator/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.474620674Z namespaces/openshift-service-ca-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.474748877Z namespaces/openshift-service-ca-operator/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.474923593Z namespaces/openshift-service-ca-operator/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.474977614Z namespaces/openshift-service-ca-operator/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.475007035Z namespaces/openshift-service-ca-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.475071177Z namespaces/openshift-service-ca-operator/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.475103988Z namespaces/openshift-service-ca-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.475152069Z namespaces/openshift-service-ca-operator/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.47518035Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.475283033Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.475363626Z namespaces/openshift-service-ca-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.475425758Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.475476599Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.475531461Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.475569902Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/service-ca-operator.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.475629174Z namespaces/openshift-service-ca-operator/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.475658885Z namespaces/openshift-service-ca-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.475715436Z namespaces/openshift-service-ca-operator/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.475722406Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.475761818Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.475916782Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.47617072Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.476376296Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.476568812Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.476748547Z namespaces/openshift-service-ca-operator/pods/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.476755868Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7695fc6c8c-2mz4v/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.476807379Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7695fc6c8c-2mz4v/service-ca-operator-7695fc6c8c-2mz4v.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.476889092Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7695fc6c8c-2mz4v/service-ca-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.476896382Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7695fc6c8c-2mz4v/service-ca-operator/service-ca-operator/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.476900352Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7695fc6c8c-2mz4v/service-ca-operator/service-ca-operator/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.476936433Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7695fc6c8c-2mz4v/service-ca-operator/service-ca-operator/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.477101388Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7695fc6c8c-2mz4v/service-ca-operator/service-ca-operator/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.477276043Z namespaces/openshift-service-ca-operator/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.477313864Z namespaces/openshift-service-ca-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.477370736Z namespaces/openshift-service-ca-operator/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.477398587Z namespaces/openshift-service-ca-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.477456988Z namespaces/openshift-service-ca/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.47751599Z namespaces/openshift-service-ca/openshift-service-ca.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.477575632Z namespaces/openshift-service-ca/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.477608303Z namespaces/openshift-service-ca/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.477664404Z namespaces/openshift-service-ca/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.477716866Z namespaces/openshift-service-ca/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.477805349Z namespaces/openshift-service-ca/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.477918452Z namespaces/openshift-service-ca/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.478016295Z namespaces/openshift-service-ca/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.478066997Z namespaces/openshift-service-ca/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.478122628Z namespaces/openshift-service-ca/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.47816902Z namespaces/openshift-service-ca/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.478204531Z namespaces/openshift-service-ca/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.478326454Z namespaces/openshift-service-ca/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.478373156Z namespaces/openshift-service-ca/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.478405487Z namespaces/openshift-service-ca/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.478493119Z namespaces/openshift-service-ca/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.478554131Z namespaces/openshift-service-ca/coordination.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.478566122Z namespaces/openshift-service-ca/coordination.k8s.io/leases/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.478598152Z namespaces/openshift-service-ca/coordination.k8s.io/leases/service-ca-controller-lock.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.478656744Z namespaces/openshift-service-ca/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.478693775Z namespaces/openshift-service-ca/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.478819349Z namespaces/openshift-service-ca/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.478896541Z namespaces/openshift-service-ca/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.479020755Z namespaces/openshift-service-ca/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.479102037Z namespaces/openshift-service-ca/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.479236761Z namespaces/openshift-service-ca/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.479377996Z namespaces/openshift-service-ca/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.479558051Z namespaces/openshift-service-ca/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.479610253Z namespaces/openshift-service-ca/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.479656664Z namespaces/openshift-service-ca/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.479689545Z namespaces/openshift-service-ca/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.479752857Z namespaces/openshift-service-ca/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.479802828Z namespaces/openshift-service-ca/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.479844789Z namespaces/openshift-service-ca/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.479931332Z namespaces/openshift-service-ca/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.480025145Z namespaces/openshift-service-ca/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.480068366Z namespaces/openshift-service-ca/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.480092727Z namespaces/openshift-service-ca/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.480168389Z namespaces/openshift-service-ca/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.48019685Z namespaces/openshift-service-ca/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.480275062Z namespaces/openshift-service-ca/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.480285523Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.480309783Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.480492439Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.480675684Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.480824699Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.481022595Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.4812098Z namespaces/openshift-service-ca/pods/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.48121723Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-r6xjw/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.481326044Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-r6xjw/service-ca-79f68b9f4f-r6xjw.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.481400186Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-r6xjw/service-ca-controller/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.481412706Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-r6xjw/service-ca-controller/service-ca-controller/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.481416827Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-r6xjw/service-ca-controller/service-ca-controller/logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.481444167Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-r6xjw/service-ca-controller/service-ca-controller/logs/current.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.481707105Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-r6xjw/service-ca-controller/service-ca-controller/logs/previous.insecure.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.481774247Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-r6xjw/service-ca-controller/service-ca-controller/logs/previous.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.481789578Z namespaces/openshift-service-ca/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.481848829Z namespaces/openshift-service-ca/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.481896701Z namespaces/openshift-service-ca/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.481925612Z namespaces/openshift-service-ca/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.481981173Z namespaces/openshift-user-workload-monitoring/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.482032725Z namespaces/openshift-user-workload-monitoring/openshift-user-workload-monitoring.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.482069896Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.482127168Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.482158769Z namespaces/openshift-user-workload-monitoring/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.48220383Z namespaces/openshift-user-workload-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.482302553Z namespaces/openshift-user-workload-monitoring/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.482383805Z namespaces/openshift-user-workload-monitoring/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.482458468Z namespaces/openshift-user-workload-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.482511619Z namespaces/openshift-user-workload-monitoring/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.48254649Z namespaces/openshift-user-workload-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.482610272Z namespaces/openshift-user-workload-monitoring/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.482649493Z namespaces/openshift-user-workload-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.482730466Z namespaces/openshift-user-workload-monitoring/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.482776897Z namespaces/openshift-user-workload-monitoring/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.482822798Z namespaces/openshift-user-workload-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.482900161Z namespaces/openshift-user-workload-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.482949312Z namespaces/openshift-user-workload-monitoring/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.482991294Z namespaces/openshift-user-workload-monitoring/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.483088886Z namespaces/openshift-user-workload-monitoring/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.483166889Z namespaces/openshift-user-workload-monitoring/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.483280592Z namespaces/openshift-user-workload-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.483353484Z namespaces/openshift-user-workload-monitoring/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.483433017Z namespaces/openshift-user-workload-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.483567611Z namespaces/openshift-user-workload-monitoring/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.483741516Z namespaces/openshift-user-workload-monitoring/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.483790757Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.483835369Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.48388456Z namespaces/openshift-user-workload-monitoring/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.483942112Z namespaces/openshift-user-workload-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.483990123Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.484025434Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.484111657Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.484190129Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.484244001Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.484302393Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.484358424Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.484415446Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.484450497Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.484456987Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.484508359Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.484663043Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.484858669Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.485010784Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.48523885Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.485448617Z namespaces/openshift-user-workload-monitoring/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.485480978Z namespaces/openshift-user-workload-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.485533729Z namespaces/openshift-user-workload-monitoring/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.48556254Z namespaces/openshift-user-workload-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.485624862Z namespaces/openshift/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.485659983Z namespaces/openshift/openshift.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.485717695Z namespaces/openshift/apps.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.485745176Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.485803767Z namespaces/openshift/apps/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.485844799Z namespaces/openshift/apps/daemonsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.485928311Z namespaces/openshift/apps/deployments.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.486006944Z namespaces/openshift/apps/replicasets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.486088046Z namespaces/openshift/apps/statefulsets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.486129677Z namespaces/openshift/autoscaling/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.486167908Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.486200879Z namespaces/openshift/batch/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.486270231Z namespaces/openshift/batch/cronjobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.486352584Z namespaces/openshift/batch/jobs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.486394475Z namespaces/openshift/build.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.486505798Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.486583011Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.486617972Z namespaces/openshift/core/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.486672173Z namespaces/openshift/core/configmaps.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.486766836Z namespaces/openshift/core/endpoints.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.486842808Z namespaces/openshift/core/events.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.486922651Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.487003953Z namespaces/openshift/core/pods.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.487084456Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.487201509Z namespaces/openshift/core/secrets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.487381734Z namespaces/openshift/core/services.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.487414446Z namespaces/openshift/discovery.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.487473307Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.487501408Z namespaces/openshift/image.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.487545269Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.488839138Z namespaces/openshift/image.openshift.io/imagestreams/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.488878209Z namespaces/openshift/image.openshift.io/imagestreams/cli-artifacts.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.488977412Z namespaces/openshift/image.openshift.io/imagestreams/cli.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.489059555Z namespaces/openshift/image.openshift.io/imagestreams/dotnet-runtime.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.489168708Z namespaces/openshift/image.openshift.io/imagestreams/dotnet.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.489343023Z namespaces/openshift/image.openshift.io/imagestreams/driver-toolkit.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.489432216Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift-java11.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.48958392Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.489770166Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java-openshift.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.48989326Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java11-openshift.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.489986642Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift-jdk11.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.490081745Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.490201689Z namespaces/openshift/image.openshift.io/imagestreams/golang.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.490326422Z namespaces/openshift/image.openshift.io/imagestreams/httpd.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.490430996Z namespaces/openshift/image.openshift.io/imagestreams/installer-artifacts.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.490517178Z namespaces/openshift/image.openshift.io/imagestreams/installer.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.490605721Z namespaces/openshift/image.openshift.io/imagestreams/java-runtime.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.490701824Z namespaces/openshift/image.openshift.io/imagestreams/java.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.490819377Z namespaces/openshift/image.openshift.io/imagestreams/jboss-datagrid73-openshift.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.49093636Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-openshift.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.491039894Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.491138386Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-openshift.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.49124347Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.491346293Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-openshift.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.491446626Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.491545209Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-openshift.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.491707743Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-runtime-openshift.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.491812207Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk11-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.491908129Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk8-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.492027973Z namespaces/openshift/image.openshift.io/imagestreams/jenkins-agent-base.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.492130306Z namespaces/openshift/image.openshift.io/imagestreams/jenkins.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.49226756Z namespaces/openshift/image.openshift.io/imagestreams/mariadb.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.492397624Z namespaces/openshift/image.openshift.io/imagestreams/must-gather.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.492485007Z namespaces/openshift/image.openshift.io/imagestreams/mysql.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.49259263Z namespaces/openshift/image.openshift.io/imagestreams/network-tools.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.492691123Z namespaces/openshift/image.openshift.io/imagestreams/nginx.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.492819707Z namespaces/openshift/image.openshift.io/imagestreams/nodejs.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.492974791Z namespaces/openshift/image.openshift.io/imagestreams/oauth-proxy.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.493067584Z namespaces/openshift/image.openshift.io/imagestreams/openjdk-11-rhel7.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.493195398Z namespaces/openshift/image.openshift.io/imagestreams/perl.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.493331842Z namespaces/openshift/image.openshift.io/imagestreams/php.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.493445766Z namespaces/openshift/image.openshift.io/imagestreams/postgresql.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.49359197Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso75-openshift-rhel8.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.493685803Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso76-openshift-rhel8.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.493796726Z namespaces/openshift/image.openshift.io/imagestreams/python.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.49393095Z namespaces/openshift/image.openshift.io/imagestreams/redhat-openjdk18-openshift.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.494074514Z namespaces/openshift/image.openshift.io/imagestreams/redis.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.494196508Z namespaces/openshift/image.openshift.io/imagestreams/ruby.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.494326752Z namespaces/openshift/image.openshift.io/imagestreams/sso75-openshift-rhel8.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.494439275Z namespaces/openshift/image.openshift.io/imagestreams/sso76-openshift-rhel8.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.494535968Z namespaces/openshift/image.openshift.io/imagestreams/tests.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.494641671Z namespaces/openshift/image.openshift.io/imagestreams/tools.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.494762405Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11-runtime.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.494898329Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.495030703Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17-runtime.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.495138476Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.49528899Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21-runtime.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.495386604Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.495493587Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8-runtime.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.49561481Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.495713583Z namespaces/openshift/k8s.ovn.org/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.495752184Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.495843297Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.4959368Z namespaces/openshift/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.495985201Z namespaces/openshift/monitoring.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.496032013Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.496080724Z namespaces/openshift/networking.k8s.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.496134996Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.496179537Z namespaces/openshift/operators.coreos.com/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.496186167Z namespaces/openshift/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.496248839Z namespaces/openshift/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.496418014Z namespaces/openshift/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.49662397Z namespaces/openshift/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.496771785Z namespaces/openshift/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.496972571Z namespaces/openshift/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.497217308Z namespaces/openshift/policy/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.497332971Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.497378393Z namespaces/openshift/route.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.497425584Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.497462335Z namespaces/openshift/template.openshift.io/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.497468475Z namespaces/openshift/template.openshift.io/templates/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.497518317Z namespaces/openshift/template.openshift.io/templates/cache-service.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.49761699Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-example.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.497737523Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-persistent.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.497848107Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-example.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.49795635Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-persistent.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.498120485Z namespaces/openshift/template.openshift.io/templates/datagrid-service.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.498239669Z namespaces/openshift/template.openshift.io/templates/django-psql-example.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.498361992Z namespaces/openshift/template.openshift.io/templates/django-psql-persistent.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.498479116Z namespaces/openshift/template.openshift.io/templates/eap-xp3-basic-s2i.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.498601179Z namespaces/openshift/template.openshift.io/templates/eap-xp4-basic-s2i.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.498712332Z namespaces/openshift/template.openshift.io/templates/eap74-basic-s2i.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.498826256Z namespaces/openshift/template.openshift.io/templates/eap74-https-s2i.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.498945859Z namespaces/openshift/template.openshift.io/templates/eap74-sso-s2i.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.499114024Z namespaces/openshift/template.openshift.io/templates/httpd-example.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.499218618Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral-monitored.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.499358602Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.499465085Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent-monitored.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.499576518Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.499683672Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.499791375Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.499899048Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.499997461Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.500107124Z namespaces/openshift/template.openshift.io/templates/mariadb-ephemeral.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.500210807Z namespaces/openshift/template.openshift.io/templates/mariadb-persistent.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.500330981Z namespaces/openshift/template.openshift.io/templates/mysql-ephemeral.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.500432544Z namespaces/openshift/template.openshift.io/templates/mysql-persistent.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.500536027Z namespaces/openshift/template.openshift.io/templates/nginx-example.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.50063948Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-example.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.500752493Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-persistent.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.500876417Z namespaces/openshift/template.openshift.io/templates/openjdk-web-basic-s2i.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.50097837Z namespaces/openshift/template.openshift.io/templates/postgresql-ephemeral.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.501081113Z namespaces/openshift/template.openshift.io/templates/postgresql-persistent.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.501179136Z namespaces/openshift/template.openshift.io/templates/rails-pgsql-persistent.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.501333751Z namespaces/openshift/template.openshift.io/templates/rails-postgresql-example.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.501454705Z namespaces/openshift/template.openshift.io/templates/react-web-app-example.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.501558328Z namespaces/openshift/template.openshift.io/templates/redis-ephemeral.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.501660891Z namespaces/openshift/template.openshift.io/templates/redis-persistent.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.501776054Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-rest-3scale.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.501894048Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-xml.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.502002111Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.502107294Z namespaces/openshift/template.openshift.io/templates/sso75-https.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.502241988Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-https.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.502371172Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.502495185Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql-persistent.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.50264052Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.502771474Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-https.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.502923578Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql-persistent.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.503070923Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.50330507Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-https.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.503431903Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.503570928Z network_logs/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.503604078Z network_logs/cluster_scale [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.503693351Z network_logs/ippools.whereabouts.cni.cncf.io [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.503767794Z network_logs/multi-networkpolicy [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.503845346Z network_logs/net-attach-def [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.503929228Z network_logs/overlappingrangeipreservations.whereabouts.cni.cncf.io [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.504079973Z network_logs/ovn_kubernetes_top_pods [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.504159825Z network_logs/ovnk_database_store.tar.gz [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.506746062Z nodes/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.506770313Z nodes/debug [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.506867726Z nodes/ip-10-0-128-215.ec2.internal/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.506916977Z nodes/ip-10-0-128-215.ec2.internal/cpu_affinities.json [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.50733678Z nodes/ip-10-0-128-215.ec2.internal/dmesg [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.507637499Z nodes/ip-10-0-128-215.ec2.internal/ethtool_channels [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.507713431Z nodes/ip-10-0-128-215.ec2.internal/ethtool_features [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.507803204Z nodes/ip-10-0-128-215.ec2.internal/ip-10-0-128-215.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.508246807Z nodes/ip-10-0-128-215.ec2.internal/irq_affinities.json [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.508427402Z nodes/ip-10-0-128-215.ec2.internal/lscpu [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.508551306Z nodes/ip-10-0-128-215.ec2.internal/lspci [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.508649219Z nodes/ip-10-0-128-215.ec2.internal/podresources.json [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.508755102Z nodes/ip-10-0-128-215.ec2.internal/pods_info.json [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.508873416Z nodes/ip-10-0-128-215.ec2.internal/proc_cmdline [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.508959778Z nodes/ip-10-0-128-215.ec2.internal/sysinfo.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.510628788Z nodes/ip-10-0-128-215.ec2.internal/sysinfo.tgz [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.510794793Z nodes/ip-10-0-131-203.ec2.internal/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.510838965Z nodes/ip-10-0-131-203.ec2.internal/cpu_affinities.json [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.511085532Z nodes/ip-10-0-131-203.ec2.internal/dmesg [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.511341829Z nodes/ip-10-0-131-203.ec2.internal/ethtool_channels [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.511415832Z nodes/ip-10-0-131-203.ec2.internal/ethtool_features [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.511507074Z nodes/ip-10-0-131-203.ec2.internal/ip-10-0-131-203.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.511978328Z nodes/ip-10-0-131-203.ec2.internal/irq_affinities.json [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.512120113Z nodes/ip-10-0-131-203.ec2.internal/lscpu [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.512245167Z nodes/ip-10-0-131-203.ec2.internal/lspci [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.51236632Z nodes/ip-10-0-131-203.ec2.internal/podresources.json [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.512468043Z nodes/ip-10-0-131-203.ec2.internal/pods_info.json [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.512579946Z nodes/ip-10-0-131-203.ec2.internal/proc_cmdline [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.512666059Z nodes/ip-10-0-131-203.ec2.internal/sysinfo.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.514466603Z nodes/ip-10-0-131-203.ec2.internal/sysinfo.tgz [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.514619897Z nodes/ip-10-0-139-134.ec2.internal/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.514652488Z nodes/ip-10-0-139-134.ec2.internal/cpu_affinities.json [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.514964337Z nodes/ip-10-0-139-134.ec2.internal/dmesg [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.515246706Z nodes/ip-10-0-139-134.ec2.internal/ethtool_channels [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.515335529Z nodes/ip-10-0-139-134.ec2.internal/ethtool_features [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.515410301Z nodes/ip-10-0-139-134.ec2.internal/ip-10-0-139-134.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.515839994Z nodes/ip-10-0-139-134.ec2.internal/irq_affinities.json [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.515980268Z nodes/ip-10-0-139-134.ec2.internal/lscpu [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.516103102Z nodes/ip-10-0-139-134.ec2.internal/lspci [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.516195034Z nodes/ip-10-0-139-134.ec2.internal/podresources.json [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.516328378Z nodes/ip-10-0-139-134.ec2.internal/pods_info.json [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.516444432Z nodes/ip-10-0-139-134.ec2.internal/proc_cmdline [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.516526784Z nodes/ip-10-0-139-134.ec2.internal/sysinfo.log [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.518213125Z nodes/ip-10-0-139-134.ec2.internal/sysinfo.tgz [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.51840017Z pod_network_connectivity_check/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.518418611Z pod_network_connectivity_check/podnetworkconnectivitychecks.yaml [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.518544684Z static-pods/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.518553895Z static-pods/kube-apiserver/ [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.52846728Z [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.528485021Z sent 71,762 bytes received 8,691,904 bytes 5,842,444.00 bytes/sec [must-gather] [must-gather-b2w4x] OUT 2026-07-13T20:25:25.528490281Z total size is 102,443,429 speedup is 11.69 [must-gather] [must-gather ] OUT 2026-07-13T20:25:25.854544501Z namespace/openshift-must-gather-s6wcw deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 79ef62dc-8bf7-4a50-8cc9-d8dfbff73685 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.24" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [git-push-artifacts] WORK_DIR: /workspace/odh-ci-artifacts [git-push-artifacts] REPO_PATH: opendatahub-io/odh-build-metadata [git-push-artifacts] REPO_BRANCH: ci-artifacts [git-push-artifacts] SPARSE_FILE_PATH: test-artifacts/docs [git-push-artifacts] SOURCE_PATH: /workspace/artifacts-dir [git-push-artifacts] DEST_PATH: test-artifacts/kserve-group-test-59prj [git-push-artifacts] ALWAYS_PASS: false [git-push-artifacts] configuring gh token [git-push-artifacts] taking github token from Konflux bot [git-push-artifacts] Initialized empty Git repository in /workspace/odh-ci-artifacts/.git/ [git-push-artifacts] Using partial fetch with sparse checkout for: test-artifacts/docs [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] * [new branch] ci-artifacts -> origin/ci-artifacts [git-push-artifacts] Already on 'ci-artifacts' [git-push-artifacts] branch 'ci-artifacts' set up to track 'origin/ci-artifacts'. [git-push-artifacts] TASK_NAME=kserve-group-test-59prj-e2e-llm-inference-service [git-push-artifacts] PIPELINERUN_NAME=kserve-group-test-59prj [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] -rw-r--r--. 1 root 1001540000 16950772 Jul 13 20:27 /workspace/odh-ci-artifacts/test-artifacts/kserve-group-test-59prj/e2e-llm-inference-service.tar.gz [git-push-artifacts] [ci-artifacts 381b7c2] Updating CI Artifacts in e2e-llm-inference-service [git-push-artifacts] 1 file changed, 0 insertions(+), 0 deletions(-) [git-push-artifacts] create mode 100644 test-artifacts/kserve-group-test-59prj/e2e-llm-inference-service.tar.gz [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] To https://github.com/opendatahub-io/odh-build-metadata.git [git-push-artifacts] 1b561a2..381b7c2 ci-artifacts -> ci-artifacts [fail-if-needed] Failing pipeline because deploy-and-e2e step failed container step-fail-if-needed has failed : [{"key":"StartedAt","value":"2026-07-13T20:27:07.675Z","type":3}]